This is the third of a series of savefile-related changes.
This adds early-days experimental support for a completely optional
'sfctool' utility (savefile conversion tool), to be able to export
a savefile's contents into a more portable format. There are likely
to be bugs at this stage. In this initial first-attempt, the export
format is a very simple ascii output.
NetHack can be built entirely, without also building this tool.
NetHack has no dependencies on the tool.
Attempts were made to minimize duplication of existing NetHack code.
To achieve that, unfortunately, #ifdef SFCTOOL and #ifndef SFCTOOL
had to be sprinkled around through some of the existing NetHack
source code, so that it could be re-used for building the utility.
The process for building the sfctool typically recompiles the source
files with #define SFCTOOL and a distinct object file with SF- is
produced.
sfctool notes:
Universal ctags is used and required to produce the sfctool utility.
Some targets were added to the Unix and Windows Makefiles to
facilitate the build process.
make sfctool
That should build a copy in util.
Note: At present, the Unix Makefiles do not copy sfctool over to the
NetHack playground during 'make install' or 'make update'.
Until that gets resolved by someone, The tool will
have to be manually copied there by the builder/admin if
desired.
cp util/sfctool ~/nh/install/games/lib/nethackdir/sfctool
Also, a separate Visual Studio sfctool.sln solution was written and
placed in sys/windows/vs. That has has only very limited testing.
Usage:
i) To convert an existing savefile to an exportascii format
that co-resides with the savefile:
sfctool -c savefile
That *must* be executed on the same platform / architecture /
data model that produced the save file in the first place.
ii) To unconvert an existing exportascii format export file to a
historical format savefile that can then be used by NetHack:
sfctool -u savefile
That must be executed on the same target platform / architecture /
data model that was used to build the NetHack that will
utilize the save file that results.
A Windows example:
sfctool -c Fred.NetHack-saved-game
That should result in creation of Fred.NetHack-saved-game.exportascii
from existing savefile:
%USERPROFILE%\AppData\Local\NetHack\3.7\Fred.NetHack-saved-game
A Unix example:
sfctool -c 1000wizard
That should result in creation of 1000wizard.exportascii.gz
from existing savefile in the playground save directory:
1000wizard.gz
Current Mechanics:
1. Makefile recipe, or script uses universal ctags to produce
util/sf.tags.
2. util/sftags is built and executed to read util/sf.tags and
generate: include/sfproto.h and src/sfdata.c.
3. util/sfctool is built from the following:
generated file compiled with -DSFCTOOL:
src/sfdata.c -> sfdata.o
existing files compiled with -DSFCTOOL:
util/sfctool.c -> sfctool.o
util/sfexpasc.c -> sfexpasc.o
src/alloc.c -> sf-alloc.o
src/monst.c -> sf-monst.o
src/objects.c -> sf-objects.o
src/sfbase.c -> sfbase.o
src/sfstruct.c -> sfstruct.o
src/nhlua.c -> sf-nhlua.o
util/panic.c -> panic.o
src/date.c -> sf-date.o
src/decl.c -> sf-decl.o
src/artifact.c -> sf-artifact.o
src/dungeon.c -> sf-dungeon.o
src/end.c -> sf-end.o
src/engrave.c -> sf-engrave.o
src/cfgfiles.c -> sf-cfgfiles.o
src/files.c -> sf-files.o
src/light.c -> sf-light.o
src/mdlib.c -> sf-mdlib.o
src/mkmaze.c -> sf-mkmaze.o
src/mkroom.c -> sf-mkroom.o
src/o_init.c -> sf-o_init.o
src/region.c -> sf-region.o
src/restore.c -> sf-restore.o
src/rumors.c -> sf-rumors.o
src/sys.c -> sf-sys.o
src/timeout.c -> sf-timeout.o
src/track.c -> sf-track.o
src/version.c -> sf-version.o
src/worm.c -> sf-worm.o
src/strutil.c -> strutil.o
This is the second of a series of changes related to save/restore.
No EDITLEVEL bump has been included, because although the code
is changed extensively by this, the content of the savefiles have
not been changed.
Push the use of the structlevel bwrite() and mread() function use
out of the core and into sfstruct.c. This is groundwork for upcoming
changes.
In the core, replace the bwrite() and mread() calls with the
use of type-specific savefile output (Sfo) and savefile
input (Sfi) macros. The macros are defined in a new header file
savefile.h, which also contains the prototypes for the sfo_* and
sfi_* functions that the macros ultimately expand to. The functions
themselves are in src/sfbase.c.
On C99, each Sfo or Sfi macro expansion refers directly to the
corresponding type-specific sfo_* or sfi_* function.
If C23 or later is is use, the majority (all but 3 types) of the
macros refer to a single _Generic output routine sfo(nhfp, dt, tag),
and a single _Generic input routine sfi(nhfp, dt, tag), which handles
the dispatch of the type-specific underlying functions. This was
somewhat experimental, but turned out to be practical because the
compiler would gripe if the type for a variable was not included in
the _Generic when passed as an argument, so it could be fixed.
This alters the savefile verication process by having a common set
return values for the related functions such as uptodate(),
check_version(), etc. The new return values return more information
about savefile incompatibilities, beyond failure/sucess. The
additional information will be useful for an upcoming addition.
The expanded return values are:
SF_UPTODATE (0) everything matched and looks good
SF_OUTDATED (1) savefile is outdated
SF_CRITICAL_BYTE_COUNT_MISMATCH (2) critical size count mismatch
SF_DM_IL32LLP64_ON_ILP32LL64 (3) Windows x64 savefile on x86
SF_DM_I32LP64_ON_ILP32LL64 (4) Unix 64 savefile on x86
SF_DM_ILP32LL64_ON_I32LP64 (5) x86 savefile on Unix 64
SF_DM_ILP32LL64_ON_IL32LLP64 (6) x86 savefile on Windows x64
SF_DM_I32LP64_ON_IL32LLP64 (7) Unix 64 savefile on Windows x64
SF_DM_IL32LLP64_ON_I32LP64 (8) Windows x64 savefile on Unix 64
SF_DM_MISMATCH (9) some other mismatch
The callers in the core have been adjusted to deal with the expanded
return values.
Other miscellaneous inclusions:
- go.oracle_loc -> svo.oracle_loc.
- add a bit (1UL << 30) to called SFCTOOL_BIT as groundwork
for changes to follow.
This is the first of several savefile-related changes to
follow later. This one is groundwork for those later changes.
Remove internal compression schemes (RLECOMP and ZEROCOMP)
and discard the savefile_info struct that was primarily used to
convey which internal compression schemes had been in use.
Relocate some struct definitions into appropriate header files
for use by code to come in later changes.
Remove the two struct size-related fields from version_info and
from the nmakedefs_s. Instead, include a series of bytes near the
beginning of the savefile, representing the size of each
struct or base data type that impacts the historical savefile
content. Those are referred to as the "critical bytes".
(Related note: the "you" struct required two bytes, low and high,
due to its size).
Compare those critical bytes in a savefile against the NetHack
build that is reading the savefile. This allows mismatch detection
early in the savefile-reading process, and a clean exit, rather than
proceeding to read nonsensical values from the file. Include some
feedback on what the first mismatch was when encountering
one.
For arrays stored in the savefile, use loop-logic in the core
to write/read the array elements one at a time, rather than in
a single blob. This will be required for changes to follow later.
(impacts artiexist[], artidisco[], svd.dungeons[], svl.level_info[],
svl.level.locations[][], msrooms[] field of mapseen, svb.bases[],
svb.disco[] objects[], svm.mvitals[], svs.spl_book[], svd.doors[],
go.oracle_loc[], utrack[], wgrowtime[])
This also adds data model to the long version information.
This invalidates existing save and bones files due to the changes in
the information at the start of the file.
I don't think this solves the recent light source reports,
but it changes a couple of things in an attempt to get more
information.
1. Having gy.youmonst.m_id field always be zero makes it tough
to distinguish it from uninitialized memory, or a random memory
value. This changes the m_id for the hero's gy.youmonst.m_id
to always hold the identifier 1, instead of 0.
2. write_ls was taking the stashed pointer in the light source,
and using it to immediately extract the m_id field and search
for that m_id. This changes the approach slightly, to actually
try and locate the stashed pointer itself in one of the monster
chains. Only if the monster pointer is located, do we dereference
it to obtain the m_id field.
3. For the interim, mark the saved ls with another set bit when
there has been a failure to locate the monst. At this time,
no code is acting on that bit, but it can be seen in a debug
session.
Hopefully, the next report will provide enough information to
understand the scenario a little better.
I remembered some more of the light source problem that led to the
panic() call that was fixed earlier today, but I'm not sure how
accurate this new comment is. If it is accurate, the changes of
impossible() to panic() that the fix included won't necessarily
catch the old, apparently one-off, problem it describes.
Reported directly to devteam: restoring a save file which was
made while the hero was polymorphed into a light emitting monster
would trigger a panic.
This was caused by an attempt to deal with corrupted save data,
which in turn was caused by attempting to use impossible() in a
situation where the game can't reliably continue. If bad light
source data was ignored during restore, it would cause trouble
during next save.
Remove the check which was erroneously detecting invalid data
and also change two impossible() calls to panic().
The static analyzer complained about use of 'obj' maybe being Null
when used in an impossible warning, but that warning will never
appear for the case where obj is actually Null. Add an assert()
that should let it figure that out, and move the impossible check
inside the 'else' clause where the check matters. (Either of those
by itself ought to be adequate to pacify the analyzer.)
Undefine some macros when the file that uses them is done so that
they won't be seen by any other source files if combined into one
huge source file. I only looked at the few files where an #undef was
needed, not all the files, but in those few files I used #undef for
[almost] all their local macros instead of just the troublesome one.
display.c is the exception; it still has lots of macros which persist
through end of file. nhlobj.c is another exception; I misremembered
the fixup for lua's lobject.c at the time and decided to include the
one #undef for nhlobj.c anyway even though 'onefile' isn't affected.
monst.c includes some reformatting. display.c's sign() macro was
redone; it's intended for efficiency compared to calling hacklib.c's
sgn() function so streamline it.
[Keni, most of the file-specific #undef fixups in genonefile.pl can
now be removed. It'll still need one for lua source file lobject.c;
addstr() there conflicts with curses.h, not with nethack's own code.]
The consolidation of global variables from scattered source
files into decl.c and declared in decl.h was begun in 3.7.0.
Their placement in common files was done for centralized
initialization and potential re-initialization during a
"play again" scenario.
It wasn't really necessary for all of them to be housed in a
single huge structure to meet the "play again" requirement,
and the single huge structure has been a little unwieldy when
it comes to maintenance.
Following this commit, instead of one single extremely large structure
named 'g' to house all of the relocated global variables, they
are distributed into several ga through gz.
To make things easy for the developer, each variable is placed
into the struct corresponding to the starting letter of the variable.
That way, no lookup is required in order to know which struct houses
a particular variable, it is a simple match to the starting letter
for all the centralized global variables.
A global variable named 'amulets', would be found in ga.
ga.amulets
^ ^
A global varable named 'move', would be found in gm.
gm.moves
^ ^
A global variable named 'val_for_n_or_more' would be found in gv.
gv.val_for_n_or_more
^ ^
A global variable named 'youmonst' would be found in gy.
gy.youmonst
^ ^
viz_array[][] is indexed by coordinates but the data it contains has
nothing to do with them so it shouldn't have been changed to coordxy.
'char' was sufficient; 'uchar' would have been better; this invents
'seenV' instead. This led to a cascade of required changes. The
result is warning free and seems to be working but my fingers are
crosssed....
One of the drivers of this change was that screen coordinates require a
type that can hold values greater than 127. Parameters to the window
port routines require a large type in order to be able to have values
a fair bit larger than COLNO and ROWNO passed to them, particularly for
their use to the right of the map window.
This splits the uses of xchar into 3 different situations, and adjusts
their type and size:
xchar
|
-----------------------
| | |
coordxy xint16 xint8
coordxy: Actual x or y coordinates for various things (moved to 16-bits).
xint16: Same data size as coordxy, but for non-coordinate use (16-bits).
xint8: There are only a few use cases initially, where it was very
plain to see that the variable could remain as 8-bits, rather
than be bumped to 16-bits. There are probably more such cases
that could be changed after additional review.
Note: This first changed all xchar variables to coordxy. Some were
reviewed and got changed to xint16 or xint8 when it became apparent that
their usage was not for coordinates.
This increments EDITLEVEL in patchlevel.h
Instead of returning 0 or 1, we'll now use ECMD_OK or ECMD_TURN.
These have the same meaning as the hardcoded numbers; ECMD_TURN
means the command uses a turn.
In future, could add eg. a flag denoting "user cancelled command"
or "command failed", and should clear eg. the cmdq.
Mostly this was simply replacing return values with the defines
in the extended commands, so hopefully I didn't break anything.
Add two new monsters and two new objects:
gold dragon
baby gold dragon
gold dragon scale mail
set of gold dragon scales
A couple of variants seem to have added these already, but this came
off my ancient list of monsters to add and was done from scratch.
It's a clone of silver dragon, but instead of having reflection and
breathing cold, a gold dragon emits light and breathes fire; because
of the latter it can be seen with infravision like a red dragon.
Adult gold dragons are lawful as in the AD&D Monster Manual rather
than chaotic as the wiki pages show for the variant versions.
Worn gold dragon scales operate similar to wielded Sunsword: when
blessed, radius is 3 (same as a lamp), if uncursed, radius is 2, and
if cursed, radius is 1 (but functions as 2 when worn by the hero,
otherwise there would be no tangible effect). Gold dragon scale mail
gets an extra +1, making blessed gold DSM have a bigger radius than
lamps. Embedded scales have radius 1 regardless of BUC state; light
for that case comes from the gold dragon monster form the hero is in.
When not worn, gold scales and scale-mail don't emit any light.
The tiles use a mix of yellow (for gold) and red. The two object
tiles seem reasonable variations of the corresponding silver dragon
ones. The two monster tiles definitely need work since the silver
ones were mostly cyan and changing that to red did not produce very
good result; subsequent attempt at a mixture was haphazard at best.
Whitelist all the verified existing triggers:
makedefs.c: In function ‘name_file’
attrib.c: one compiler balks at a ? b : c for fmtstring
cmd.c: In function ‘extcmd_via_menu’
cmd.c: In function ‘wiz_levltyp_legend’
do.c: In function ‘goto_level’
do_name.c: In function ‘coord_desc’
dungeon.c: In function ‘overview_stats’
eat.c: one compiler balks at a ? b : c for fmtstring
end.c: one compiler balks at a ? b : c for fmtstring
engrave.c: In function ‘engr_stats’
hack:c one compiler balks at a ? b : c for fmtstring
hacklib.c: one compiler balks at a ? b : c for fmtstring
insight.c: one compiler balks at a ? b : c for fmtstring
invent.c: In function ‘let_to_name’
light.c: In function ‘light_stats’
mhitm.c: In function ‘missmm’
options.c: In function ‘handler_symset’
options.c: In function ‘basic_menu_colors’
options.c: In function ‘optfn_o_autopickup_exceptions’
options.c: In function ‘optfn_o_menu_colors’
options.c: In function ‘optfn_o_message_types’
options.c: In function ‘optfn_o_status_cond’
options.c: In function ‘optfn_o_status_hilites’
options.c: In function ‘doset’
options.c: In function ‘doset_add_menu’
options.c: In function ‘show_menu_controls’
options.c: In function ‘handle_add_list_remove’
pager.c: In function ‘do_supplemental_info’
pager.c: In function ‘dohelp’
region.c: In function ‘region_stats’
rumors.c: sscanf usage
sounds.c: In function ‘domonnoise’
spell.c: In function ‘dospellmenu’
timeout.c: In function ‘timer_stats’
topten.c: In function ‘outentry’, fscanf, sscanf, fprintf usage
windows.c: In function ‘genl_status_update’
zap.c: one compiler balks at a ? b : c for fmtstring
win/curses/cursstat.c: In function ‘curses_status_update’
win/tty/wintty.c: In function ‘tty_status_update’
win/win32/mswproc.c: In function ‘mswin_status_update’
If we ever want huge maps with COLNO or ROWNO larger than signed char,
this will at least allow the game to compile and start when typedef'ing
xchar to int. Trying to use huge maps exposes more bugs.
ignitable() was excluding magic lamp and then every place that
used it did so as 'ignitable(obj) || obj->otyp == MAGIC_LAMP'
so just include magic lamp.
I noticed that while hunting for an explanation for report #K2734
where returning to a previously visited level triggered the
warning "begin_burn: unexpected eggs". I've decided that the
zombie apocalypse is probably the cause. It inserted a new type
of timer in the list of such but it didn't bump EDITLEVEL to
invalidate save and bones files which relied on indices into the
old list. I'm not sure whether we should bump that now.
Don't replace a monster that's been temporarily seen via camera
flash or thrown/kicked lit candle/lamp with "unseen, remembered
monster" glyph if it can be sensed via telepathy, warning, or
extended monster detection.
Implement the suggested feature that a camera's flash actually update
hero's memory of the map as it traverses across the level. Turned
out to be more work than anticipated despite having the code for a
thrown or kicked lit candle or lamp to build upon.
Among other things it needed to update the circle code to handle
previously unused radius 0 to operate on the center point only. I've
never touched that before and hope this hasn't introduced any bugs.
Also removes several instances of vision code operating on column #0.
(At least one is still present.)
After some discussion with Alex Smith, it seems like a good change for
both gameplay and realism that candles' light radius should decay
quadratically instead of exponentially. Now a light radius of 4 from
candles can be accomplished by burning 9 candles, and players who find a
lot of candles might even be able to get up to 5 (16 candles) or 6 (25
candles).
The main impetus for this change is that with the existing formula, the
more candles -> more light mechanic was more or less useless outside of
wizard mode, because you needed 49 candles to do better than a lamp.
Make some progress on a couple of next minor release checklist
items, hopefully without introducing too many new bugs. This
is just the initial commit, and work continues.
Checklist items:
Savefiles compatible between Windows versions, whether 64-bit
or 32-bit in little-endian field format.
Selection of file formats:
historical (structlevel saves),
lendian (little-endian, fieldlevel saves),
and just for proof-of-concept, ascii fieldlevel saves
(the ascii is huge! 10x bigger than little-endian).
For the fieldlevel save, all complex data structures recursively
get broken down until until it is one of the simple types that
can't be broken down any further, and that gets when it gets
written to the output file.
New files needed for this build:
hand-coded:
include/sfprocs.h
src/sfbase.c - really a dispatcher to one of the
output/input format routines.
src/sflendian.c - little-endian output writer/reader.
src/sfascii.c - ascii text output writer/reader.
auto-coded (generated):
include/sfproto.h
src/sfdata.c
This is just one approach. I'm sure there are countless others
and they have different pros and cons.
For producing the auto-coded files a utility called
universal-ctags, that is actively maintained and evolving,
was used to do all the heavy-lifting of parsing the
NetHack C sources to tabulate the data fields, and store
them in an intermediate file called util/nethack.tags
(not required for building NetHack if you already have a
generated include/sfproto.h and src/sfdata.c)
util/readtags (also not required for building NetHack
itself) will decipher the nethack.tags file and produce
the functions that can deal with the NetHack struct data
fields.
You can obtain the source for universal-ctags by cloning it
from here:
https://github.com/universal-ctags/ctags.git
The combination universal-ctags + util/readtags has been
tried and tested under both Windows and Linux, so it is
not tied to a particular platform.
Note: util/readtags will work only with universal-ctags
output, so other ctags are unlikely to work as-is.
Universal-ctags can be build from source very easily
under Linux, or under Windows using visual studio.
Throwing or kicking a lit lamp, lit candle, or lit potion of oil
wasn't giving off any light as it travelled to its destination.
Now it does, and dungeon features, objects, or monsters that are
temporarily seen as it moves from square to square till appear on
the map. In the monster case, they go away as soon as the light
moves beyond range, but when it finishes moving the "remembered,
unseen monster" glyph will be drawn at their location. I think that
part has some room for improvement, but mapping temporarily seen
terrain features is the primary impetus for this change.
Also, any message delivery while the "lit missile" travelled still
showed its light around the hero. Noticeable for lamps or stacks
of sufficient candles if hero has no other light source.
This cannibalizes the monst->mburied bit for temporarily seeing a
monster. It has been present but unused for ages. I needed to
replace a couple of vision macros to make sure they didn't examine
it any more so that overloading for transient lighting doesn't
introduce any vision oddities. For version $NEXT, monst->mtemplit
can be given its own bit. It is only set during bhit() execution
and cleared by the time that returns, so has no effect on save files.
