9. Release Notes¶
This following summarizes the most important changes in recent Spicy releases. For an exhaustive list of all changes, see the CHANGES file coming with the distribution.
9.1. Version 1.2¶
New Functionality
GH-913: Add support for switch-level
&parse-at
and&parse-from
attributes inside a unit.Add optimizer pass removing unimplemented functions and methods.
This introduces a global pass triggered after all individual input ASTs have been finalized, but before we generate any C++ code. We then strip out any unimplemented member functions (typically Spicy hooks), both their definitions as well as their uses.
In order to correctly handle previously generated C++ files which might have been generated with different optimization settings, we disallow optimizations if we detect that a C++ input file was generated by us.
Changed Functionality
- Add validation of unit switch attributes. We previously silently ignored unsupported attributes; now errors are raised.
- Remove configure option
--build-zeek-plugin
. Spicy no longer supports building the Zeek plugin/analyzers in-tree. This used to be available primarily for development purposes, but became challenging to maintain. - Add environment variable HILTI_CXX_INCLUDE_DIRS to specify additional C++ include directories when compiling generated code.
- GH-940: Add runtime check for parsing progress during loops.
Bug fixes
- Fix computation of unset locations.
- Fix accidental truncating conversion in integer code.
Documentation
9.2. Version 1.1¶
New Functionality
GH-844: Add support for
&size
attribute to unitswitch
statement.GH-26: Add
%skip
,%skip-pre
and%skip-post
properties for skipping input matching a regular expression before any further input processing takes place.Extend library functionality provided by the
spicy
module:crc32_init()/crc32_add()
compute CRC32 checksums.mktime()
creates atime
value from individual components.zlib_init()
initializes aZlibStream
with a given window bits argument.Zlib
now accepts a window bits parameter.
Add a new
find()
method to units for that searches for abytes
sequence inside their input data, forward or backward from a given starting position.Add support for
&chunked
when parsing bytes data with&until
or&until_including
.Add
encode()
method tostring
for conversion tobytes
.Extend parsing of
void
fields:- Add support for
&eod
to skip all data until the end of the current input is encountered. - Add support for
&until
to skip all data until a deliminator is encountered. The deliminator will be extracted from the stream before continuing.
- Add support for
Port Spicy to Apple silicon.
Add Dockerfile for OpenSUSE 15.2.
Changed Functionality
- Reject
void
fields with names. - Lower minimum required Python version to 3.2.
- GH-882: Lower minimum required Bison version to 3.0.
Bug fixes
- GH-872: Fix missing normalization of enum label IDs.
- GH-878: Fix casting integers to enums.
- GH-889: Fix hook handling for anonymous void fields.
- GH-901: Fix type resolution bug in
&convert
. - Fix handling of
&size
attribute for anonymous void fields. - Fix missing update to input position before running
%done
hook. - Add validation rejecting
$$
in hooks not supporting it. - Make sure container sizes are runtime integers.
- Fix missing operator<< for enums when generating debug code.
- GH-917: Default-initialize forwarding fields without type arguments.
Documentation
- GH-37: Add documentation on how to skip data with
void
fields.
9.3. Migrating from the old prototype¶
Below we summarize language changes in Spicy compared to the original research prototype. Note that some of the prototype’s more advanced functionality has not yet been ported to the new code base; see the corresponding list on GitHub for what’s still missing.
Changes:
Renamed
export
linkage topublic
.Renamed
%byteorder
property to%byte-order
.Renamed
&byteorder
attribute to&byte-order
.Renamed
&bitorder
attribute to&bit-order
.All unit-level properties now need to conclude with a semicolon (e.g.,
%random-access;
).Renamed
&length
attribute to&size
.Renamed
&until_including
attribute to&until-including
.Replaced
&parse
with separate&parse-from
(taking a “bytes” instance) and&parse-at
(taking a stream iterator) attributes.Attributes no longer accept their arguments in parentheses, it now must
<attr>=expr
. (Before, both versions were accepted.)uint<N>
andint<N>
are no longer accepted, useuintN/intN
instead (which worked before already as well)list<T>
is no longer supported, usevector<T>
instead.New syntax for parsing sequences: Use
x: int8[5]
instead ofx: vector<int8> &length=5
. For lists of unknown size, usex: int8[]
. When parsing sequences sub-units, use:x: Item[]
; or, if further arguments/attributes are required,x: (Item(1,2,3))[]
. (The latter syntax isn’t great, but the old syntax was ambiguous.)New syntax for functions:
function f(<params>) [: <result>]
instead of<result> f(<params>)
Renamed runtime support module from
Spicy
tospicy
(so useimport spicy
)In units, variables are now initialized to default values by default. Previously, that was (inconsistently) happening only for variables of type sink. To revert to the old behaviour, add “&optional” to the variable.
Renamed type
double
toreal
.Generally, types don’t coerce implicitly to bool anymore except in specific language contexts, such as in statements with boolean conditions.
Units using any of the random access methods (e.g.,
input()
), now need to explicitly add a unit property %random-access.Filters can now be implemented in Spicy itself. The pre-built
filter::Base64Decode
andfilter::Zlib
provide the base64 and zlib functionality of the previously built-in filters.{unit,sink}::add_filter
are renamed to{unit,sink}::connect_filter
.Enums don’t coerce to bool anymore, need to manually compare to
Undef
.Coercion to bool now happens only in certain contexts, like
if
-conditions (similar to C++).The sink method
sequence
has been renamed tosequence_number
.The effect of the sink method
set_initial_sequence_number
no longer persists when reconnecting a different unit to a sink.&transient
is no longer a supported unit field attribute. The same effect can now be achieved through an anonymous field (also see next point).$$
can now be generally used in hooks to refer to the just parsed value. That’s particularly useful inside hooks for anonymous fields, including fields that previously were&transient
(see above). Previously, “$$” worked only for container elements inforeach
hooks (which still operates the same way).Fields of type
real
are parsed with&type
attribute (e.g.,&type=Spicy::RealType::IEEE754_Double
). They used to&precision
attributes with a different enum type.Assigning to unit fields and variables no longer triggers any hooks. That also means that hooks are generally no longer supported for variables (This is tricky to implement, not clear it’s worth the effort.)
When importing modules, module names are now case-sensitive.
When parsing vectors/lists of integers of a given length, use
&count
instead of&length
.Zeek plugin:
Bro::dpd_confirm()
has been renamed tozeek::confirm_protocol()
. There’s also a correspondingzeek::reject_protocol()
.- To auto-export enums to Zeek, they need to be declared public.