Related Posts
Additional Posts in Data & Analytics Consultants
Bowl Leader
Thought this was interesting. Across 160 teams of researchers, just about all failed to make good life outcome predictions on things like GPA, evictions, layoffs, and others. Data followed 4.5k families across 15 years, with 13k features (varied over time). Haven't looked at it directly yet, but will be turning the docs and data inside out... In the meantime, authors claim this as showing the limits of ML. Oh, and it's published in PNAS, so you know there's some big publication energy there.
https://www.pnas.org/content/117/15/8398
Bowl Leader
New to Fishbowl?
Download the Fishbowl app to
unlock all discussions on Fishbowl.
unlock all discussions on Fishbowl.
Coach
I've done this with Python for cleaning and analyzing data when we do the same process each time, creating standardized deliverables (in Excel or Word), and things where I'm getting info from the web in a look up/copy/past kind of way via web scraping, and updating graphs automatically when the data changes. But they really have to be standardized for this to work, or else you're writing in so much processing for each possible contingency that it's faster not to automate.
And to really automate, you're getting it off your computer and onto a server where you set it to run regularly either by time (via cron) or by an event based trigger (like lambda), which requires that your stuff (like your data) also be accessible by that server. Otherwise you're still running the scripts manually. Although that can still be worth it.
Identify the obnoxiously specific steps you take in a task, then find a way to code each one. Like DS1 said, you can templetize most stuff this way, if not outright automate. Most you can do with a plain ol local script. Search for coding language of choice + tool to be automated + "generation" as a starting point. Use prompts or forms for user inputs that can't be automated. Before you know it, you're building specialized software.