Neutron Universe

Homelab Diagram

Tue, 09 Dec 2025 08:00:00 GMT

flowchart TB
  %% WAN
  Internet(("Internet\n1 Gbps full duplex"))
  Router["GL.iNet GL-MT6000\nWireGuard + AdGuard Home"]
  Switch["TP-Link TL-SG105S-M2\n2.5G switch"]

  Internet -->|1 Gbps| Router
  Router -->|2.5 Gbps| Switch

  %% Hosts
  MiniPC["Mini PC\nmicrok8s node\n1 Gbps"]
  FullPC["Server + GPU\nmicrok8s node\n2.5 Gbps"]
  TrueNAS["TrueNAS NAS\n2x2 mirror\n~32 TiB usable"]

  Router -->|1 Gbps| MiniPC
  Switch -->|2.5 Gbps| FullPC
  Switch -->|2.5 Gbps| TrueNAS

  %% TrueNAS services
  subgraph "TrueNAS services"
    MinIO["MinIO S3"]
    Shares["SMB / NFS shares"]
  end

  TrueNAS --> MinIO
  TrueNAS --> Shares

  %% microk8s clusters
  subgraph "Server microk8s"
    FullK8s["MoneyMachine / OpenWebUI / MM\nIngress + CF tunnels"]
  end

  subgraph "Mini PC microk8s"
    MiniK8s["Infra / monitoring\nCF tunnels"]
  end

  FullPC --> FullK8s
  MiniPC --> MiniK8s

  %% External access
  subgraph "External access"
    CF["Cloudflare tunnels / Zero Trust"]
    WGClients["WireGuard clients\nlaptops, phone, etc"]
  end

  FullK8s --> CF
  MiniK8s --> CF
  CF --> Internet
  Router -->|WireGuard| WGClients

flowchart TB
  subgraph "TrueNAS"
    TN[TrueNAS 32 TiB]
    TN --> MinIO[(MinIO S3)]
    TN --> SMB[SMB shares]
    TN --> NFS[NFS exports]
  end

  subgraph "Server microk8s"
    NSmm[namespace: neutron-mm]
    NSmedia[namespace: neutron-media]

    Selastic[Elasticsearch]
    Skibana[Kibana]
    Semby[Emby]
    Snextcloud[Nextcloud]
  end

  subgraph "Mini PC microk8s"
    NSinfra[namespace: neutron-infra]
    Scloudflared[cloudflared tunnels]
    Smonitoring[monitoring / observability]
  end

  NSmm --> Selastic
  NSmm --> Skibana
  NSmedia --> Semby
  NSmedia --> Snextcloud

  NSinfra --> Scloudflared
  NSinfra --> Smonitoring

Wikipedia Deep Research

Fri, 03 Oct 2025 07:00:00 GMT

Project Overview

Realizing that there exists a majority of “Deep Research” tasks stemming from summarizing non-realtime components, I figured I could reduce research costs significantly by using the open sourced deep research from LangGraph and a locally hosted Wikipedia instance for retrieving topic context.

I setup a hosted Wikipedia site and “Open Deep Research” (bundled with Wikipedia embeddings) over FastAPI service to handle requests inside of my local microk8s cluster.

This took quite a bit of effort to setup/get working and, given my limited free time, it will take me another bit of effort to stand up a repo/submit a PR to the LangGraph repo for stable, portable code. Forgive me for not including the entire change code.

Goals

Offline retrieval: no external web search calls during research besides LLM calls
Semantic search over Wikipedia via txtai embeddings
Deterministic citations that point to your local Kiwix URLs from latest Wikipedia ZIM dump
Drop-in tool named web_search so the supervisor/researcher loops reuse the same interface.

Dependencies

LangGraph Open Deep Research
txtai embeddings for finding similar articles to an input query
Kiwix hosted Wikipedia site using a recent ZIM dump

High-level architecture

LLM plan -> researcher
		-> `web_search` tool
        -> `wiki_search_async(queries)`
        -> txtai Embeddings
        -> result doc ids
        -> Kiwix-serve HTTP fetch (local ZIM)
        -> HTML
        -> Markdown sanitizer
        -> structured “SOURCE i” blocks with local URLs
        -> compression
        -> final report

Key Kubernetes components

Kiwix mirror: kiwix-serve on Kubernetes, fronted by Ingress at https://wiki.<domain> with mounted ZIM volume
Embeddings store and Research API: txtai with neuml/txtai-wikipedia used when calling the Deep Research wiki_search search tool when setting the configuration search_api=WIKIPEDIA

Wikipedia Search details

Query resolution
wiki_search_async fans out queries. Each query runs a txtai similar(...) SQL against the Wikipedia index. Keep hits above a percentile threshold (default 0.99) to avoid low-signal pages
Document fetch
For each hit, format the Wikipedia title and fetch content from Kiwix:
- Try direct https://wiki.<ingress_domain>/content/<ZIM>/content/A/<Title>
- Else resolve via https://wiki.<ingress_domain>/content/<ZIM>/search?pattern=...&books.name=<ZIM> and follow the content link
Normalization
Strip HTML, remove references section, collapse whitespace, and output concise Markdown for model-friendly summarization
Citations
Emit SOURCE i blocks with the canonical local Kiwix URL:
```
https://wiki.<domain>/content/<ZIM>/A/<Title>
```
Synthesis
The normal compression and final-report nodes run unchanged. The only difference is the source of notes.

Integration with other search providers

No changes to planners or reducers. The tool registry wires wiki_search as the canonical web_search when search_api=WIKIPEDIA. You can still swap to Tavily or Exa by changing the config without touching graph logic.

Security and reproducibility

All fetches remain inside the cluster.
Results are pinned to your ZIM snapshot for auditability.
Ingress TLS terminates at your controller using your wildcard cert.

Cost Difference

Costs vary from my semi-structured testing (run report, check OpenAI usage cost difference as measurement) but both Open Deep Research instances use the same default LLM configuration (see the README.md) and just isolating whether using local/OpenAI Web Search significantly decreases cost per report

Research Provider	Search Provider	Report Cost
Open Deep Research	OpenAI Web Search	$2.36
Open Deep Research	Wikipedia	$0.32

Summary

This adds a local, deterministic, citation-clean retrieval path for the Deep Research agent. It keeps the LangGraph contract the same, swaps only the search backend, and produces final reports with reliable, linkable local sources.

Final note: if you’re still looking for an external provider, from using several “Deep Research” providers, it seems like the lowest dollar cost is OpenAI’s o4-mini-deep-research-2025-06-26 model

Example Wikipedia Report

Query: Write a company overview report on NVIDIA, noting the key suppliers/customers/input materials/ and industry/sector dynamics

NVIDIA Company Overview Report (October 2025)

Company Overview

Corporation, headquartered in Santa Clara, California, is a global leader in accelerated computing and artificial intelligence (AI). Since its founding in 1993, NVIDIA has evolved from its original focus on graphics processing units (GPUs) for gaming into a multifaceted technology powerhouse developing solutions for data centers, AI, professional visualization, autonomous vehicles, and edge computing.

For the fiscal year ending January 26, 2025, NVIDIA commands an overwhelming 92% of the discrete GPU market for desktops and laptops. Its principal business areas are:

Gaming: GeForce GPUs remain industry standards for gaming and creative workloads.
Data Center and AI: Serving hyperscale cloud providers, enterprise AI workloads, and scientific research.
Professional Visualization: Supplying graphics hardware and software to professionals, including designers and researchers.
Automotive and Embedded Systems: Providing system-on-chip (SoC) and AI solutions for driver assistance, autonomous vehicles, and robotics.

NVIDIA’s product development is organized around advanced GPU architectures (e.g., Ada Lovelace, Hopper, and Blackwell), and its solutions increasingly blend hardware with proprietary software stacks, like CUDA for high-performance computing and the Omniverse platform for 3D collaboration and simulation [1].

Recent strategic moves have deepened NVIDIA’s position in generative AI and cloud infrastructure, and expanded its ecosystem through acquisitions and partnerships with leading data center, automotive, and cloud companies. Subsidiaries, such as Mellanox Technologies (high-speed datacenter networking) and Cumulus Networks (software-defined networking), bolster its vertical integration strategy [1].

Key Suppliers

Major Suppliers and Their Roles

NVIDIA follows a “fabless” semiconductor model, focusing on design while outsourcing manufacturing and major component production to third parties. The company’s crucial suppliers include:

Semiconductor Foundries:
- Taiwan Semiconductor Manufacturing Company (TSMC): The principal foundry partner for NVIDIA’s most advanced GPUs and AI processors. TSMC fabricates chips using leading-edge process nodes (e.g., 4nm, 5nm), which is essential for keeping NVIDIA’s products competitive in terms of performance and energy efficiency [2].
- Samsung: Occasionally serves as a secondary foundry partner for select GPU lines, providing manufacturing diversification and capacity flexibility.
Memory Suppliers:
- SK Hynix, Samsung, Micron: Supply high-bandwidth memory (HBM), GDDR6/6X DRAM, and other specialized memory critical for GPU and AI workloads.
Substrate and Packaging Providers:
- Unimicron, Ibiden, and others: Produce advanced substrates essential for high-performance, multi-die GPU assemblies.
Component Suppliers:
- Several companies provide capacitors, power management ICs, and connectors. Although individual firms are rarely named in disclosures, NVIDIA notes a dependence on a broad network of specialized component makers.
Software Partners:
- Microsoft: Collaborates with NVIDIA to integrate support for features like real-time ray tracing (RTX) with DirectX APIs, broadening NVIDIA’s platform reach [3].

Supplier Importance

TSMC’s role is critical—any disruption in TSMC’s advanced process node capacity would have immediate repercussions for NVIDIA’s product roadmap and supply chain resilience [2].
Memory suppliers are also vital, especially as advanced AI and GPU workloads require ever more bandwidth and capacity, making these relationships strategic for product competitiveness.
Substrate shortages have, in industry-wide fashion, periodically constrained GPU and AI accelerator shipments, emphasizing the importance of securing reliable substrate vendor partnerships.

Limitations

Detailed supplier lists (by name, proportion, or spend) are confidential and not comprehensively disclosed publicly outside of general industry awareness and historic referencing in filings [1][2]. The analysis above reflects best-available industry insights.

Major Customers

Customer Segments and Examples

NVIDIA’s products serve a widely diversified global customer base, but revenue and business influence are concentrated in a few main segments:

Cloud Service Providers (“Hyperscalers”): The largest customers by far are global cloud operators who deploy NVIDIA hardware for AI training, inference, and cloud-based GPU/AI compute services. These include industry giants—such as Amazon Web Services (AWS), Microsoft Azure, Google Cloud, and, to a lesser extent, Alibaba and Tencent—who each deploy tens or hundreds of thousands of GPUs in their data centers.
Enterprise and High-Performance Computing (HPC): Large enterprises, universities, and national laboratories leverage NVIDIA hardware for AI research, simulations, and large-scale computation.
Gaming OEMs and Channel Partners: PC manufacturers, gaming system builders, and distributors who bundle GeForce GPUs into consumer systems.
Automotive: Leading automotive OEMs and Tier-1 suppliers source NVIDIA’s DRIVE compute platforms for driver assistance and autonomous vehicle programs.
Government and Research: National projects related to AI, supercomputing, and scientific research.

Customer Concentration and Analysis

The data center segment—driven by AI and cloud computing—accounted for the majority of NVIDIA’s recent revenue growth as of 2024–2025.
Specific customer identities are typically not disclosed in SEC filings for confidentiality. However, filings sometimes reference the risk of customer concentration, and it is well-understood that a handful of cloud hyperscalers account for a significant percentage of total revenue in data center sales.
Gaming remains a critical revenue base but is more diversified across OEM and channel partners, while the automotive and enterprise sectors show moderate but growing concentration.

Limitations

Precise revenue assignments to specific named companies are generally not available due to customer privacy or contractual restrictions. Where public, the main hyperscalers are widely inferred as dominant clients through industry reporting and partner announcements.

Critical Input Materials

Manufacturing of NVIDIA’s GPUs and AI accelerators is underpinned by a variety of sophisticated materials and components, which are critical to product performance and supply chain continuity.

Key Materials

Semiconductor Wafers (Silicon): Advanced wafers—particularly 300 mm—manufactured at nodes as fine as 4 nm or below, primarily provided by TSMC.
Advanced Packaging and Substrates: High-end, multi-layer substrates capable of handling power, heat, and multi-die interconnection (e.g., CoWoS, HBM interposers), produced by companies such as Unimicron and Ibiden.
High-Bandwidth Memory (HBM, GDDR6/6X): Provided by SK Hynix, Samsung, and Micron. Used for rapid data access and parallel processing critical to AI workloads.
Rare Earth Elements and Specialty Metals: Materials like cobalt, tantalum, gold, palladium, and various rare earths are used in tiny amounts in semiconductor fabrication and board components.
Specialty Chemicals and Gases: Used in lithography, etching, deposition, and cleaning processes during chip production (e.g., photoresists, gases like argon and fluorine).

Supply Chain Considerations and Challenges

Supply of advanced substrates and memory has proven to be a chokepoint, with industry-wide constraints affecting not just NVIDIA but all advanced semiconductor vendors.
Geopolitical risks (e.g., US-China trade tensions, export controls on advanced tech to China) have prompted NVIDIA to redesign certain SKUs and adapt its sourcing in response to regulatory restrictions.
The scale and complexity of AI hardware have increased per-unit material requirements, putting further pressure on upstream suppliers and making supply chain management a major focus for NVIDIA and its partners.

Limitations

Granular breakdowns of supplier-specific sourcing by material quantity or value are not published for competitive and commercial sensitivity reasons. Broad categories and major vendor names are industry facts supported by public manufacturing announcements and supply chain analyses.

Industry and Sector Dynamics

Competitive Landscape

NVIDIA operates within several overlapping yet competitive sectors:

AI and Accelerated Computing: Competitors include AMD (Instinct and Radeon), Intel (Xe accelerators, Gaudi AI processors), and rapidly scaling custom AI chip ventures by hyperscalers themselves (e.g., Google TPU, Amazon Trainium).
Semiconductors and GPUs: NVIDIA maintains dominance in discrete GPUs but faces constant pressure from AMD in gaming and professional visualization.
Networking and Interconnect: Mellanox (a NVIDIA business), Intel, Broadcom, and Marvell compete to define next-generation datacenter fabrics and storage interconnects.

Major Trends (2024–2025)

AI Boom and AI Infrastructure Buildout: Unprecedented demand for AI training/inference hardware driven by advances in large language models, generative AI, and enterprise AI adoption. Investments such as the proposed Stargate LLC (a $500 billion+ US AI infrastructure plan) dwarf historic data center buildouts, setting new demand records [5].
Edge and Automotive Expansion: AI and video processing are pushing computational intelligence to the edge and into vehicles; NVIDIA’s automotive platforms power critical ADAS and autonomous test fleets worldwide.
Supply Chain and Geopolitics: Global tension has led to export restrictions, especially around sales of advanced AI chips to China, causing NVIDIA to adapt supply and design strategies to remain compliant yet competitive [4].
Industry Consolidation and Ecosystem Building: Mergers, acquisitions, and tighter software-hardware integration are defining the next wave of competitive differentiation.

Challenges

Supply Chain Volatility: Shortages of substrates, wafers, and memory chips can limit NVIDIA’s ability to fulfill demand.
Regulatory Risks: Export bans (notably to China) and technology transfer restrictions are persistent threats.
Customer Concentration: Dependence on a small number of cloud hyperscaler customers exposes NVIDIA to cyclical procurement and market power risks.
Talent and R&D Pressure: The global race for AI technical talent and the escalating costs of advanced chip R&D create barriers for smaller rivals but drive up operational costs.

Industry Outlook

The intersection of AI, cloud computing, and high-performance hardware cements NVIDIA’s leading sector position as of October 2025, with market momentum but significant risks in supply chains, regulatory frameworks, and competitive rivalry.

Sources

Code Agent Prompt

While I am not including the code changes (formatting/submitting the PRs would take time and I’m already exhausted from a day job), I have pasted the necessary files into ChatGPT to generate a prompt for a coding agent to recreate the code as a starting template inside of a freshly cloned Open Deep Research repo if you want to implement this project here

OCR LLM Librarian

Fri, 30 May 2025 07:00:00 GMT

Introduction

I recently decided to digitize the books I’ve read the last 10 years with the intent of making the content searchable so I don’t have to spend time going through the index/remembering what ideas were on which pages to help better solidify my understanding and also to have LLM recommended next books to read based on the content of my reading history, not just the title and related books.

The project breaks down into three layers:

Acquisition: photography of every page
Understanding: LLM-powered OCR that outputs semantically-rich JSON (HTML, figures, highlights, footnotes, page numbers…)
Retrieval-Augmented Generation (RAG): an Elasticsearch vector index plus an Open WebUI function that lets me chat with my entire library

Only displaying some of the unique steps here as the rest of the code is pretty standard from quickstarts for Elasticsearch and llama index

Camera setup

I used my old Canon Rebel T5i fix mounted to my desk along with an LED light connected to my desktop using a Mini USB to USB-C cable and gphoto in bash to automate a timer taking a picture every 6 seconds which allowed me enough time to flip through each page of the book which outputs to a book-specific directory in an input subdirectory

gphoto2 --capture-image-and-download --force-overwrite --filename "$DEST/%Y%m%d_%H%M%S.%C" --interval 6

OpenAI Batch OCR

Used OpenAI’s gpt-4o-mini model plus batch requests to send each image with a structured Pydantic schema to fill out containing the HTML of the page content, figures/images, and highlighted text

Stage	Key function(s)	What happens
1. Pre-process	`crop`, `split`	Perspective-correct, crop both pages, detect centre gutter and save left/right images.
2. Batch OCR	`run_batch_ocr` → OpenAI Batch API	Each page is encoded to base-64, wrapped in a `BookOCR` Pydantic schema, and submitted in ~170 MB JSONL chunks.
3. Post-process	`read_batch_response`, `parse_objects`	Responses are validated back into Pydantic, objects/figures are cropped out and saved, token costs are tallied.

BookOCR pydantic model that forces well-formed HTML and image placeholders (<img src="Figure 3-2">), so later scripts can inline saved crops

class Object(BaseModel):
 
    name: str = Field(description='The object name, sometimes denoted by just a single word/phrase or the bold word "Figure" and a hierarchical number like "Figure 1.x.x"', default='')
    description: str = Field(description='The object description (if available) of text below the name.', default='')
    bounding_box: BoundingBox = Field(description='Bounding box coordinates of the object.')
    index: int = Field(description='Index integer starting from 0 at the top of the page and increasing downwards', default=-1)
 
 
class Highlight(BaseModel):
 
    text: str = Field(description='The color highlighted text', default='')
 
 
class BookOCR(BaseModel):
 
    html: str = Field(description='Formatted output HTML of the image as a single text string', default='')
    objects: list[Object] = Field(description='List of objects in the image. This includes images, figures, or subsections.',default=[])
    content_text: str = Field(description='Text content of the main content without any other captions or references or footnotes', default='')
    highlights: list[Highlight] = Field(description='List of color highlighted text phrases or sentences often in colors such as yellow, orange, pink, blue, etc.', default=[])
    footnotes: str = Field(description='Text content of the footnotes (near bottom of page) content without any other captions or references or footnotes', default='')
    page_number: int = Field(description='Optional if found integer page number of the book', default=None)

System prompt:

SYSTEM_MSG = """
You are an expert OCR and layout analyst. When processing an image, use the defined JSON schemas (BoundingBox, Object, Highlight and BookOCR) as guidance for your structured output. Your output must be a well-formed HTML fragment (including <html> and <body> tags) that preserves headings, paragraphs, lists, tables, images, figures, and figure captions.
 
Extract images or any figures by obtaining their name, caption, bounding box, and page offset index. For each detection, output only the bounding box information in the format <x1><y1><x2><y2>. If the figure/image is only text, do not output a Object.
 
When inserting images into the HTML, tag them using the <img> element with a placeholder for the src attribute. Instead of a generic placeholder, incorporate the Object.name value so that each <img> tag looks like <img src="{Object.name}" alt="{Object.name}"/>. This links the parsed objects/images by their common name in the HTML, enabling downstream browsers to correctly load the images.
 
Extract any highlighted text phrases or sentences.
"""

Elasticsearch Vectorstore

Indexed all text content from JSON documents output from OCR step to use for RAG search

For each document:

Strips text from the html field with BeautifulSoup.
Embeds with all-MiniLM-L6-v2 (384-dim).
Indexes twice:
- books – global corpus.
- books_<slug> – per-book index for scoped search.

Mappings include both a dense_vector field and redundant metadata.page_number / num_objects so RAG results can cite page numbers immediately

Open WebUI Function

Finally, for using the vectors within a chat I built an Open WebUI function that performs RAG search over the indexed book pages given a query

Uses the RetrieverQueryEngine from llama index with tree-summarize mode to turn the top-k pages into a tidy, hierarchical answer
Switch between global or single-book search by filling the book valve inside the chat
Ships as a simple plugin (plugin.json) so any model chatting in WebUI can call it without code changes

AWS Lambda Parallelizing Invocations with S3 Batch Operations

Tue, 20 Oct 2020 07:00:00 GMT

The Problem

I have a simple Lambda function that downloads historical equity trades for a given symbol (“AAPL”) and date (“2020–01–01”) and saves the output as a parquet file to S3 for analysis but how can I parallelize the invocations when my input list is very large? If I use Boto3 and Python on my local machine, the downloads of symbol+date are sequentially invoked which results in O(N) performance. Searching around online for a better solution, I found a post by Ivan Klishch in which he used a separate controller Lambda that accepted a range of input and invoked the base Lambda function in a “ fan-out ” pattern. This works pretty well but I think I figured out a better solution using S3 batch operations.

What is S3 Batch Operations?

AWS S3 Batch Operations is designed to perform large-scale batch operations on S3 objects using a manifest that can be generated automatically using a S3 inventory report. Among the batch operations that can be performed, you can specify a Lambda function to invoke which, for each invocation, the bucket and key name of an object is provided.

Batch Operations seems useful for existing S3 objects but what about objects that don’t exist yet? Can we tell Batch Operations our desired objects and let it handle the Lambda invocations? The answer is yes, yes we can.

Generating a Custom Manifest with Lambda Arguments

According to the Batch Operations console, the manifest file can be any CSV with the columns “BucketName”, “S3Key”, and an optional “Version” column.

So to generate a custom CSV for downloading historical equity trades, I would just need to generate a CSV containing the arguments for my Lambda such as:

“BucketName”,”ABBV/2020–01–02”
“BucketName”,”ABBV/2020–01–03”
“BucketName”,”ABBV/2020–01–04”
“BucketName”,”ABBV/2020–01–05”
………….etc…………..

Here I am using the “S3Key” column value as my Lambda argument list delimited by a backslash “/” but really all three columns could be used as arguments to Lambda since they are bound to the event message by Batch Operations passed to each invocation.

I use a simple Python script to loop over the desired symbols plus dates to generate the CSV (with no headers) “manifest.csv” and upload it to my S3 bucket so Batch Operations can find it:

Generating a test manifest with 1 symbol and 200 dates

Then I setup a S3 Batch Operations job and specify my custom manifest as the input:

And specify my downloader Lambda:

Configure an output directory for the Batch results including any download errors:

Then create the job config, activate it, and wait for results:

And quicker than you can say “mitochondria is the powerhouse of the cell”, the job completes and my data is available:

Conclusion

This method of using a custom input CSV of Lambda arguments lets AWS S3 Batch Operations handle all of the parallelization and is cost effective (only $0.25 to run a batch job compared to doing all scheduling/invocation manually). There are of course multiple ways of ingesting data like this and I welcome any feedback on better methods of doing so.

Realtime Stock Trade Analysis With Apache NiFi, Kafka, and Flink (and Python)

Thu, 15 Oct 2020 07:00:00 GMT

The Grand Canyon: the ultimate data flow

Introduction

In this article we will dive into a way to monitor and trade realtime stock trades using several Apache applications and Python. All resources are available on the GitHub repo. When I say realtime I mean live streaming equity trades since the concept of “realtime” is tricky to define with a time series. This pipeline is able to handle all ~9000 equities (and all other trade feeds) at once since each stage is horizontally scalable (you get what you pay for). If you aren’t familiar with the technologies involved:

NiFi: data processing application that uses a directed graph to organize a “flow” of data
Kafka: distributed log that acts as a streaming database using Producers/Consumers of messages
Flink: distributed processing engine with stateful computations
Python: Python

For the actual trading strategy, I will be using some stochastic variation functions from my own academic research that require maintaining a state of past log returns.

Data Source

To receive trade data, I use Polygon as a provider since they have full trade streaming available with WebSockets because who would ever want a message based credit system. The stock trade event schema we’ll be working with is:

Event Type: String
Symbol Ticker: String
Exchange ID: Integer
Trade ID: String
Tape: Integer (1=”A”, 2=”B”, 3=”C”)
Price: Double
Trade Size: Integer
Trade Conditions: List[String]
Trade Timestamp: Unix NS (you know, for nano seconds)

NiFi

To actually ingest the streaming trade data, I use NiFi’s ConnectWebSocket processor and an input list of equity symbols. Then I push all received trades to a Kafka instance.

Here is an overview of the flow:

The flow centers around the ConnectWebSocket processor that handles the connection, authentication, subscription, and trade message reception which then pushes messages to Kafka with topics keyed by equity symbol.

To be use this graph, simply build the custom NiFi docker image that adds the flow to the base Apache image. Also add your Polygon token to the docker-compose config so NiFi can pull data:

Kafka

I use Kafka so that I can persist and receive the messages ingested by NiFi. In the event that a NiFi producer or Flink consumer goes down, it doesn’t interrupt the rest of the system and the producer/consumer can be restarted and pull messages from where they left off using consumer offsets. Kafka relies on Zookeeper for broker metadata so we need to also add it to our docker-compose config as well:

Flink

I use Flink to subscribe to the trade messages and perform windowed aggregations. Flink is good because you can implement stateful computations with a checkpointing mechanism. Thus if we required a substantial amount of stateful information (incremental aggregations or indicators) for our trade calculations, our pipeline would be fault tolerant enough to quickly recover from any downtime. Sure I could use Spark Streaming but Flink has more complex windowing classes that would prove to be useful in the future with more complexly timed strategies (click-session windowing for equity trades??).

The Scala code for our Flink job (forgive any bad code, I’m still learning the art of Scala):

The trade metrics we calculate are:

Log of Price Change:

Quadratic Variation

Realized Variation

Bipower Variation

From a combination of these papers:

And adding Flink’s containers to the docker-compose config:

Python Agent

I use a basic Python class to implement an agent that subscribes to aggregate signals and, on message polls, updates its state and places orders. The agent also uses several subclasses to hold state about the portfolio, namely a portfolio class that has handles to several asset instances to hold current portfolio positions and calculate stats. For the broker, I’m using Alpaca since they have $0 commission equity trades and access to a sandbox API with a free account on sign-up.

On each message poll to Kafka (given a polling frequency that can be altered to fit how often the agent should trade), the agent updates its internal state, ranks its portfolio, generates new equity weights, and submits orders to rebalance. Here is the same thing I just said but in box form:

An interesting feature of this design is that one could have N number of agents each implementing a different portfolio, ranking strategy, and/or update frequency. A result of fully decoupling each layer in the pipeline, each component can be specialized and tuned.

Once you have an account, add the API key and secret to the docker-compose config under the agent.environment key so the agent has access:

Change APCA_URL based on Paper/Live trading

And set the Dockerfile config with these steps:

Yes I know I can just use the base python image without ubuntu but future proofing for java apps

Build and Run

We need to build the submittable Flink jar application and the custom NiFi image before running the pipeline. Using a Makefile, I define two targets:

And then add the entry points for our pipeline:

Since we didn’t bundle our Flink jar, we must submit the job manually to the cluster using the submit-flink target but only after we first start all of our other services using the run target. Running make run first builds our Flink job:

And our custom NiFi image:

Run make run then, once the Kafka topics are created by NiFi, then run make submit-flink to submit the Flink job. Be sure to run this during market hours or there won’t be any trade data to pull and operate on.

NiFi will start pulling messages and pushing to their respective Kafka topics from which Flink will calculate our metrics over finally pushing the aggregates back to Kafka for the Python trading Agent to receive and trade upon.

Tmux split with “make run” on top, “make submit-flink” on bottom

Thanks for reading, I’m not that interested in over-engineering this pipeline anymore so hopefully I passed at least a minuscule crumb of knowledge onto you. I pray to the code gods that your copy and paste of this project worked perfectly the first time 🎲 (honestly there have been random encoding errors but I swear this pipeline has worked most of the time).

Neutron Universe

Homelab Diagram

Wikipedia Deep Research

Project Overview

Goals

Dependencies

High-level architecture

Wikipedia Search details

Integration with other search providers

Security and reproducibility

Cost Difference

Summary

Example Wikipedia Report

NVIDIA Company Overview Report (October 2025)

Company Overview

Key Suppliers

Major Suppliers and Their Roles

Supplier Importance

Limitations

Major Customers

Customer Segments and Examples

Customer Concentration and Analysis

Limitations

Critical Input Materials

Key Materials

Supply Chain Considerations and Challenges

Limitations

Industry and Sector Dynamics

Competitive Landscape

Major Trends (2024–2025)

Challenges

Industry Outlook

Sources

Code Agent Prompt

OCR LLM Librarian

Introduction

Camera setup

OpenAI Batch OCR

Elasticsearch Vectorstore

Open WebUI Function

AWS Lambda Parallelizing Invocations with S3 Batch Operations

The Problem

What is S3 Batch Operations?

Generating a Custom Manifest with Lambda Arguments

Conclusion

Realtime Stock Trade Analysis With Apache NiFi, Kafka, and Flink (and Python)

Introduction

Data Source

NiFi

Kafka

Flink

Python Agent

Build and Run

That’s the way the cookie crumbles