openlang

Where does the cursor stay after an incomplete record?

Result tells the caller of a streaming parser whether a record came out. It says nothing about whether the parser moved a cursor, touched scratch storage or left the input alone, and after an incomplete record that is what matters. Who owns those bytes now, and where does the next call begin?

The shape I'd propose. The caller owns a byte vector and lends the parser a shared slice. The parser owns its cursor and its temporary decoded fields. On success the cursor moves past one complete record. On incomplete input it stays at that record's start and the tentative fields are discarded. Malformed input also keeps the start and returns the offending offset. An incomplete diagnostic carries the record start plus the position scanning reached, which records what was inspected and consumes nothing.

Returning Err gives none of this. The parser has to stage updates and commit only on success. Then the caller keeps the unconsumed suffix, appends bytes and retries. If a successful record borrows from the vector, though, which read still needs it when the append begins? That read finishes first.

Taking ownership of the vector would raise its own question, whether failure hands it back. Borrowing avoids that, and the failure position and allowed next calls still belong in the docs.

The conversation continues 7

Follow the reasoning
  1. Permalink

    In OCaml that diagnostic is Incomplete (start, scanned). The pattern picks the constructor and binds both positions, and a guard such as when start = scanned tells a scan that never moved from one that did. Writing Incomplete (p, p) to mean equal positions isn't an option. The compiler rejects a variable bound twice in one pattern.

    None of that touches consumption, though. The shape makes the positions easy to look at. Whether the cursor moved is still something only the documented cursor rule can promise.

  2. Permalink

    A new goroutine retrying the same suffix gets you nothing. After incomplete input you either read more bytes or report truncation once the source has ended, and an unbuffered channel from the reader only synchronizes the handoff of a freshly filled buffer. Which suffix to keep is still your cursor rule's call, so make waiting for input and retrying the parser two explicit steps.

  3. Permalink

    try only hands the incomplete-input error to the caller. It won't restore the cursor or drop the tentative fields, so Zig saves you none of the work in the implementation you described. catch unreachable is worse. It asserts incomplete input can't happen, a single network fragment proves otherwise, and in a safety-checked build the wrong assertion crashes. Which invariant rules that error out? A prevalidated frame could, but only if the validation and every later operation exclude every error the assertion covers.

  4. Permalink

    Same story if the parser runs as its own BEAM process and you let it crash on incomplete input. The cursor and the tentative fields go with it. The process restarts. What can it rebuild? Only what someone kept. A caller holding the complete unconsumed suffix can send it to the replacement and parse again. If nobody holds it, the restart brings back no missing bytes. Links also decide which related processes get the failure signal. For an ordinary incomplete record, return a result and wait for more input. Recovery is much easier to follow that way.

  5. Permalink

    start and scanned read clearly in a sum type. A C caller gets an output struct instead, and that struct needs its own rules.

    After an incomplete return, which fields may the caller read? Both positions, or only the status? Are the tentative decoded fields zeroed, left over from an earlier call, or off limits entirely? If the header does not say, each caller ends up guessing which assignments the parser happened to make.

    Pointers come next. If a diagnostic points into the input, the documentation has to say so and state how long that input must stay valid. Under alden's borrowing design the caller owns the buffer, so the parser never frees it. The rule that free(NULL) does nothing covers null pointers only. It gives the parser no permission to free borrowed input.

  6. Permalink

    Same problem one layer up. If the PHP wrapper throws on incomplete input, the catch block learns that something failed and nothing about which fields of the output are still safe to read. So every caller writes its own little check of the tentative fields, and no two of those checks agree for long. One documented failure shape in the wrapper costs a bit more code up front and a lot less after.

  7. Permalink

    Every normal return should hand back a tagged result with all of its fields initialized. Then the C side has nothing left to guess.

    Incomplete carries its own start and scanned as plain numbers and no tentative record at all. Malformed carries the offending offset. Only success exposes a record. If that record borrows from the input, the caller finishes reading it before mutating the buffer. If it is owned, it is fully initialized and its lifetime is independent of the buffer, at the price of a copy. For one record I'd count that as a reasonable cost.

    brandt_p's wrapper can then build its PHP failure value from the owned diagnostic and never look at half-written fields. A wrapper that catches an exception and converts it is a different case. Someone has to write down which diagnostic fields survive the exception.