The Real Lesson of Automating Your Job Search (It’s Not About the Jobs)

A few days ago, a developer named Beltagy published something that stopped me mid-scroll: a thorough, brutally honest write-up of a personal automation project called job-digest—a Python pipeline that scrapes job boards, enriches listings with an LLM, ranks them, and pings a Telegram bot every morning with the top 3 matches.

It wasn’t a startup pitch. It wasn’t a SaaS waiting for product-market fit. It was the kind of project that gets built at 11 PM when someone is tired of doing the same manual task for the 73rd time and decides, I would rather write code for 20 hours than do this by hand ever again.

That’s something I deeply respect. And the story—all 2,000 words of it—taught me more about the discipline of automation than a dozen conference talks.

The grind that sparked the code

Beltagy had applied to 73 jobs in 4 weeks. Custom cover letters. CV tweaks per role. Cold LinkedIn messages. A Notion database tracking everything. The whole exhausting, performative circus.

Result: ghosted by 61, rejected by 8, phone screen with 4. That’s a 5% conversion rate. By any reasonable standard, that’s a broken process.

While studying Go for a homelab project (and dealing with a Cilium policy update that literally set the cluster on fire), they were burning out not from interviews, but from the search itself. So they did what any engineer with an itch and a terminal would do: they wrote a scraper.

I don’t know Beltagy personally. But I know the feeling. The impulse to automate isn’t always about efficiency. Sometimes it’s about regaining control over a system that treats you like a cog. And the job hunt is definitely one of those systems.

The pipeline: simple, brutal, personal

At its core, job-digest is a classic ETL pipeline:

1. Async scrapers pull listings from LinkedIn Jobs, Indeed, Glassdoor, and remote-first sites.
2. A normalizer unifies fields (location → “Amsterdam”, seniority detection, remote policy flags).
3. Hard filters knock out anything not in target cities, below Senior level, or missing the right tech keywords.
4. An LLM enricher (GPT-4o-mini) judges fit quality, scores the role 0–100, and returns a simple verdict: yes / no / maybe.
5. A composite ranker weights LLM score, salary, company tier, and recency.
6. A Telegram bot delivers the top 3 at 7 AM. Commands like /more or /details reveal the rest only on demand.

It runs on a Raspberry Pi 4 in a homelab cron job, storing data in a single SQLite file with WAL mode. No serverless, no microservices. One device, one responsibility.

And yet… this pipeline cut daily browsing time from 45 minutes to 5. It surfaced roles at Adyen, Booking.com, and N26 that the author swears they would have missed manually. It also surfaced a stealth-mode fintech, a Berlin CDN rebuilding its edge on Cilium, and a Dublin startup that sounded like a match made in heaven.

The architecture decisions that matter

Beltagy’s table of design choices is worth studying. Every decision has a clear reason, and most of them are about ruthless pragmatism, not elegance.

• Python 3.11 + asyncio over Go: rapid prototyping won. BeautifulSoup and httpx are battle-tested.
• SQLite over PostgreSQL: zero infra, portable, single file. WAL mode handles concurrent reads from the enricher and notifier.
• httpx + BeautifulSoup over Selenium/Playwright: faster, lighter, cheaper—with a headless fallback only for JavaScript-rendered boards.
• Adaptive backoff (exponential + jitter) over fixed delays: respects robots.txt without being slower than necessary.
• GPT-4o-mini over a local LLM: at ~$0.02 per 100 jobs, the quality is worth the tiny cost.
• Telegram over email: it’s where the author already lives. Push notification = instant awareness, no inbox noise.

Every choice screams “I’m building this for exactly one person.” That’s the secret. The more specific the user, the bolder the simplifications you can afford.

5 failures I’ve already stolen for my own playbook

The write-up includes a candid list of five things that broke before the pipeline worked. These aren’t “haha oops” stories; they’re sharp engineering lessons wrapped in humility.

1. LinkedIn rate-limited the scraper in 6 minutes

The initial attempt used a single httpx session with a 2-second delay. LinkedIn returned 429s after 30 requests and started serving CAPTCHAs. The fix? Stop scraping HTML. Instead, hit the public JSON endpoint at /jobs-guest/jobs/api/jobs. It’s more stable and returns structured data directly.

Don’t fight the frontend. Find the API the frontend uses.

I’ve applied that same principle a hundred times reverse-engineering undocumented internal APIs. The best scraper is the one that doesn’t look like a scraper.

2. The LLM hallucinated a role at Google that didn’t exist

GPT-4o-mini confidently invented a “Kubernetes Platform Engineer” position at Google Amsterdam with a €500K salary. The URL 404’d. The fix: every scraped URL gets a live HEAD request before enrichment. LLMs are enrichment tools, not validators.

LLMs are pattern matchers, not fact checkers.

I’ve seen security researchers get burned by this too—asking an LLM to summarize a vulnerability report and getting a CVE number that was never assigned. Always verify.

3. The author applied to the same job 4 times

The initial dedup key was (title, company). Then a company reposted the same role with a slightly different title. The fix: content hashing on the normalized description. Exact matching is naive; job boards are messy.

Same description = same job, even if the title changes.

4. Running it on a laptop meant it died when the lid closed

The first version was a manual python main.py that ran for 8 minutes. The fix: a headless Pi that runs on a cron schedule every morning at 06:00 CET. It also handles DNS, monitoring, and now job hunting.

Automations that depend on you remembering to run them are not automations.

5. Getting 18 “good” matches caused decision paralysis

Even with filtering, seeing 18 roles in one digest led to opening 12 tabs and applying to none. The fix: show top 3 by default. Commands /more and /all reveal the rest on demand. This tiny UX change turned browsing into action.

More data is not better. More decisions is worse.

That last one is a principle I’ve started applying outside of job searches. The best dashboards I’ve built don’t show everything—they show what needs attention now.

The open-source bet

The code is public at github.com/beltagyy/job-digest, and Beltagy is actively seeking collaborators. It’s a classic open-source pitch: “This works for my profile. Help me make it work for yours.” The roadmap is full of pragmatic next steps: more scrapers (Hacker News “Who is Hiring?”, Greenhouse/Workday backends), better filtering (visa sponsorship detection, experience level parsing), a local LLM option for people who don’t want OpenAI costs, a Discord bot, and a web dashboard.

I love that the project doesn’t pretend to be generic. It’s not trying to be a platform. It’s a tool that scratches one person’s itch, with a structure clean enough that others can contribute their own itches.

What this teaches us about automation (the non-technical part)

Strip away the scrapers and the Telegram calls, and there’s a deeper philosophy here. Beltagy didn’t automate the important part—they automated the boring part. The bot finds roles. The human still writes every cover letter by hand and researches every company. The pipeline handles noise; the person handles signal.

That’s the difference between a useful automation and one that just shifts busywork. Automate the discovery and the filtering, yes. But never outsource the parts that require judgment. If you’re applying for a senior platform role, the cover letter is where you demonstrate that judgment. A chatbot can’t do it for you.

Second, the tech stack taught the author something new. They already knew Python but had never built a production async pipeline. Now they understand asyncio.gather, SQLite WAL, structured LLM output, and backpressure. If your side project doesn’t stretch you, it’s just unpaid labor.

Third—and this is the one I want to shout from the rooftops—ship before it’s perfect. The first version had no LLM enrichment, no scoring, no Telegram bot. Just a list of job titles and URLs. It was ugly. It still saved 30 minutes a day. Each increment added value, but the core deliverable existed on day one.

I’ve seen too many engineers (myself included) hold back on releasing a tool because it doesn’t have a fancy dashboard or a polished README. The only thing that matters at the start is that it works for you.

The result isn’t a job offer (yet)

The author is still interviewing. But the metrics matter:

• Time spent browsing jobs: 45 minutes → 5 minutes.
• Applications per week: ~8 → ~12, consistently.
• Roles matching their stack: 30% → 85%.
• Time-to-apply after posting: 3–4 days → under 24 hours.
• Recruiter conversations went from 1–2 per month to 4–5.

Beyond the numbers, the real gain is mental bandwidth. Job searching moved from “something I do when I’m exhausted at 10 PM” to “a system that runs without me.” That’s the promise of any good automation: it doesn’t make you faster; it makes you less distracted.

Should you build your own?

If you’re actively job hunting and you know Python, absolutely. The source is modular enough that you can fork it, point it at your target cities and tech stack, and have a digest running by tomorrow morning. And if you’re not job hunting, consider what other repetitive, low-judgment task is eating your mental cycles.

Personal automation pipelines—the kind that live on a Pi or an old laptop under your desk—are some of the most satisfying software you’ll ever write. They’re not scalable. They’re not marketable. They’re a gift from present-you to future-you, and that makes them strangely pure.

I’ll be watching the job-digest repo. Not because I need a job right now, but because the lessons it embodies are the kind I want to study, steal, and share. If you’re a collaborator, I bet the author would love a PR that adds a scraper for your local board, or a clever regex to detect visa sponsorship, or a mock LLM handler for CI. It’s the kind of project that grows not by product roadmap, but by people scratching their own itches together.

Go read the original article. Clone the repo. And then look at your own daily grind and ask: what’s the one boring, soul-sucking task I could automate by next weekend? The answer might surprise you.