twilight venturesopening a large file without loading it whole
Every Lens product opens a file on its own structure instead of reading it end to end first. JSONL Lens and Log Lens index the byte offset of each line in the background while the first page is already on screen. Parquet Lens reads the row groups, record batches, blocks or records the file is made of and decodes only the ones on the page. SQLite Lens and DuckDB Lens bring up the schema tree first and page rows in as you scroll. XLSX Lens streams the sheet once and spills rows to a scratch index, Notebook Lens indexes where every cell sits, and Heap Dump Lens walks the dump in one pass with the per-object state kept on disk. Memory follows the page and the index rather than the file (the heap dump index keeps a 2.6 GB dump inside a 512 MB cap), the file is never loaded whole, and a file far beyond the IDE's own large-file limit opens the same way, in a tab beside the plain text editor.
at what sizes
| product | the file | what happens |
|---|---|---|
| Parquet Lens | a 10 GB dataset | pages the way a 10 MB one does |
| XLSX Lens | a 500,000-row sheet | the first page renders while the rest is still loading |
| Notebook Lens | a 175 MB notebook with 9,000 cells, images and long outputs | opens in about a second; cell 9,000 is one Go click away |
| Heap Dump Lens | a 2.6 GB dump, twenty million objects | indexes in seconds inside a 512 MB cap |
| SQLite Lens | a 5,000,000-row table | Find streams down every row at about 350 MB/s |
| JSONL Lens for VS Code | a 1.02 GB file of 5,000,000 records | indexes in 3.7 s, on screen while the index builds |
| Parquet Lens for VS Code | a 5,000,000-row file | opens in under ten milliseconds |
| Lens File Viewer for Jira | a 38 MB log attachment | opens in about three seconds, with level counts for the whole file |
| Lens File Viewer for Confluence | a 61 MB JSONL attachment with 400,000 events | read in small byte ranges as you page |
paging
The table viewers fetch rows in pages, up to 5000 at a time in Parquet Lens, SQLite Lens and DuckDB Lens, and each has a jump: Go to row, Go to record, Go to line, or Go to cell in Notebook Lens. No product caps the rows. A table pages to its last row, a JSONL file to its last line, a sheet to its last row, and in the VS Code extensions a 5,000,000-record file scrolls to its last record. Scrolling and paging never wait on a search: search runs on a reader of its own.
what the status line says
Under a Log Lens tab the status line reads Page 1 / 30 with the file size beside it, and every product has a line like it that says what has been read and how much there is. The ones below are read off the screenshots on the product pages.
| product | the line reads |
|---|---|
| JSONL Lens | Page 1 / 24, with the file size beside Reload, and Matches only 1 / 44 matching records once a search narrows the view |
| Log Lens | Page 1 / 30 and 264.9 KB, plain text, with the level chips reading ERROR 147, WARN 268, INFO 1,605, DEBUG 580 once the whole file is counted |
| Parquet Lens | 20,000 rows, 7 columns, 22 blocks, 568.2 KB, Avro, snappy; and Page 1 / 2,400 on a larger file |
| SQLite Lens | orders, rows 1-100 of 300000, 59.4 MB |
| XLSX Lens | 'perf', rows 1-100 of 499999 (499500 data rows) |
| Notebook Lens | 9000 cells, 6000 code, 2500 md, 500 raw, nbformat 4.5, 175.4 MB |
| DuckDB Lens | orders, rows 1-100 of 120000, 4.8 MB |
| Heap Dump Lens | service-heap.hprof, 2.45 GB, 20,030,919 objects, 725 classes |
| JSONL Lens for VS Code | 5,000,000 records, 1019.9 MB, 11 columns (sampled 2,000) |
| Parquet Lens for VS Code | 1,000,000 rows, 10 columns, 10 row groups, 18.4 MB, Parquet, snappy |
Paging and search work while the index runs, and when it finishes the counts fill in: Log Lens turns its level chips from blank to exact whole-file numbers, Find in XLSX Lens keeps a live match count while it scans, and the schema stats in JSONL Lens say which sample they came from until a Pro full-file scan swaps in exact figures.
compressed files
A .jsonl.gz or a .log.gz is asked how big it will get before it is decompressed to a temporary file, and a file that cannot fit in temp space is refused up front with both numbers named, instead of failing after minutes of work. The note under the file reads Decompressing with the file name and the temp space it needs. When the container can only prove a floor, the note says at least that much. A blocked gzip file, the kind bgzip writes, opens in place in JSONL Lens with no temp copy, and the status line says so. A .hprof.gz is inflated into a work directory of its own that goes when the tab closes. In the Jira app, a gzip-wrapped log, JSONL or CSV inflates in the browser under a 384 MB ceiling; a larger file asks for a download instead.
databases
SQLite Lens takes no lock and writes no sidecar, so a database a running app still owns is safe to open. Frames still waiting in the -wal are called out in a banner with a one-click snapshot reload that never touches the original, both free; listing those frames one per row is a Pro feature. A database pulled from a device shows its newest rows when the -wal and -shm files are copied along with it. DuckDB Lens opens the engine in its read-only mode; when another process holds the write lock, it says so and opens a snapshot copy, and you browse the copy while the writer keeps running.
in the browser
In Confluence and Jira the file stays on Atlassian and the viewer fetches it in byte ranges, as the person viewing the page or the issue. A page fetches only the slice on screen, while the record index, a whole-file search and the level scan of a log read the whole attachment in ranges. A 61 MB JSONL attachment with 400,000 events pages while its records are still being indexed; a 3.2 million row Parquet file opens on its schema and pulls row groups in as they are needed.