This is the third of a series of savefile-related changes.
This adds early-days experimental support for a completely optional
'sfctool' utility (savefile conversion tool), to be able to export
a savefile's contents into a more portable format. There are likely
to be bugs at this stage. In this initial first-attempt, the export
format is a very simple ascii output.
NetHack can be built entirely, without also building this tool.
NetHack has no dependencies on the tool.
Attempts were made to minimize duplication of existing NetHack code.
To achieve that, unfortunately, #ifdef SFCTOOL and #ifndef SFCTOOL
had to be sprinkled around through some of the existing NetHack
source code, so that it could be re-used for building the utility.
The process for building the sfctool typically recompiles the source
files with #define SFCTOOL and a distinct object file with SF- is
produced.
sfctool notes:
Universal ctags is used and required to produce the sfctool utility.
Some targets were added to the Unix and Windows Makefiles to
facilitate the build process.
make sfctool
That should build a copy in util.
Note: At present, the Unix Makefiles do not copy sfctool over to the
NetHack playground during 'make install' or 'make update'.
Until that gets resolved by someone, The tool will
have to be manually copied there by the builder/admin if
desired.
cp util/sfctool ~/nh/install/games/lib/nethackdir/sfctool
Also, a separate Visual Studio sfctool.sln solution was written and
placed in sys/windows/vs. That has has only very limited testing.
Usage:
i) To convert an existing savefile to an exportascii format
that co-resides with the savefile:
sfctool -c savefile
That *must* be executed on the same platform / architecture /
data model that produced the save file in the first place.
ii) To unconvert an existing exportascii format export file to a
historical format savefile that can then be used by NetHack:
sfctool -u savefile
That must be executed on the same target platform / architecture /
data model that was used to build the NetHack that will
utilize the save file that results.
A Windows example:
sfctool -c Fred.NetHack-saved-game
That should result in creation of Fred.NetHack-saved-game.exportascii
from existing savefile:
%USERPROFILE%\AppData\Local\NetHack\3.7\Fred.NetHack-saved-game
A Unix example:
sfctool -c 1000wizard
That should result in creation of 1000wizard.exportascii.gz
from existing savefile in the playground save directory:
1000wizard.gz
Current Mechanics:
1. Makefile recipe, or script uses universal ctags to produce
util/sf.tags.
2. util/sftags is built and executed to read util/sf.tags and
generate: include/sfproto.h and src/sfdata.c.
3. util/sfctool is built from the following:
generated file compiled with -DSFCTOOL:
src/sfdata.c -> sfdata.o
existing files compiled with -DSFCTOOL:
util/sfctool.c -> sfctool.o
util/sfexpasc.c -> sfexpasc.o
src/alloc.c -> sf-alloc.o
src/monst.c -> sf-monst.o
src/objects.c -> sf-objects.o
src/sfbase.c -> sfbase.o
src/sfstruct.c -> sfstruct.o
src/nhlua.c -> sf-nhlua.o
util/panic.c -> panic.o
src/date.c -> sf-date.o
src/decl.c -> sf-decl.o
src/artifact.c -> sf-artifact.o
src/dungeon.c -> sf-dungeon.o
src/end.c -> sf-end.o
src/engrave.c -> sf-engrave.o
src/cfgfiles.c -> sf-cfgfiles.o
src/files.c -> sf-files.o
src/light.c -> sf-light.o
src/mdlib.c -> sf-mdlib.o
src/mkmaze.c -> sf-mkmaze.o
src/mkroom.c -> sf-mkroom.o
src/o_init.c -> sf-o_init.o
src/region.c -> sf-region.o
src/restore.c -> sf-restore.o
src/rumors.c -> sf-rumors.o
src/sys.c -> sf-sys.o
src/timeout.c -> sf-timeout.o
src/track.c -> sf-track.o
src/version.c -> sf-version.o
src/worm.c -> sf-worm.o
src/strutil.c -> strutil.o
This is the second of a series of changes related to save/restore.
No EDITLEVEL bump has been included, because although the code
is changed extensively by this, the content of the savefiles have
not been changed.
Push the use of the structlevel bwrite() and mread() function use
out of the core and into sfstruct.c. This is groundwork for upcoming
changes.
In the core, replace the bwrite() and mread() calls with the
use of type-specific savefile output (Sfo) and savefile
input (Sfi) macros. The macros are defined in a new header file
savefile.h, which also contains the prototypes for the sfo_* and
sfi_* functions that the macros ultimately expand to. The functions
themselves are in src/sfbase.c.
On C99, each Sfo or Sfi macro expansion refers directly to the
corresponding type-specific sfo_* or sfi_* function.
If C23 or later is is use, the majority (all but 3 types) of the
macros refer to a single _Generic output routine sfo(nhfp, dt, tag),
and a single _Generic input routine sfi(nhfp, dt, tag), which handles
the dispatch of the type-specific underlying functions. This was
somewhat experimental, but turned out to be practical because the
compiler would gripe if the type for a variable was not included in
the _Generic when passed as an argument, so it could be fixed.
This alters the savefile verication process by having a common set
return values for the related functions such as uptodate(),
check_version(), etc. The new return values return more information
about savefile incompatibilities, beyond failure/sucess. The
additional information will be useful for an upcoming addition.
The expanded return values are:
SF_UPTODATE (0) everything matched and looks good
SF_OUTDATED (1) savefile is outdated
SF_CRITICAL_BYTE_COUNT_MISMATCH (2) critical size count mismatch
SF_DM_IL32LLP64_ON_ILP32LL64 (3) Windows x64 savefile on x86
SF_DM_I32LP64_ON_ILP32LL64 (4) Unix 64 savefile on x86
SF_DM_ILP32LL64_ON_I32LP64 (5) x86 savefile on Unix 64
SF_DM_ILP32LL64_ON_IL32LLP64 (6) x86 savefile on Windows x64
SF_DM_I32LP64_ON_IL32LLP64 (7) Unix 64 savefile on Windows x64
SF_DM_IL32LLP64_ON_I32LP64 (8) Windows x64 savefile on Unix 64
SF_DM_MISMATCH (9) some other mismatch
The callers in the core have been adjusted to deal with the expanded
return values.
Other miscellaneous inclusions:
- go.oracle_loc -> svo.oracle_loc.
- add a bit (1UL << 30) to called SFCTOOL_BIT as groundwork
for changes to follow.
It seems surpristing that no one has noticed this since the code that
is responsible has been present for six months. Inventory list at
end of game included bogus "? - (list likely candidates)".
This is the first of several savefile-related changes to
follow later. This one is groundwork for those later changes.
Remove internal compression schemes (RLECOMP and ZEROCOMP)
and discard the savefile_info struct that was primarily used to
convey which internal compression schemes had been in use.
Relocate some struct definitions into appropriate header files
for use by code to come in later changes.
Remove the two struct size-related fields from version_info and
from the nmakedefs_s. Instead, include a series of bytes near the
beginning of the savefile, representing the size of each
struct or base data type that impacts the historical savefile
content. Those are referred to as the "critical bytes".
(Related note: the "you" struct required two bytes, low and high,
due to its size).
Compare those critical bytes in a savefile against the NetHack
build that is reading the savefile. This allows mismatch detection
early in the savefile-reading process, and a clean exit, rather than
proceeding to read nonsensical values from the file. Include some
feedback on what the first mismatch was when encountering
one.
For arrays stored in the savefile, use loop-logic in the core
to write/read the array elements one at a time, rather than in
a single blob. This will be required for changes to follow later.
(impacts artiexist[], artidisco[], svd.dungeons[], svl.level_info[],
svl.level.locations[][], msrooms[] field of mapseen, svb.bases[],
svb.disco[] objects[], svm.mvitals[], svs.spl_book[], svd.doors[],
go.oracle_loc[], utrack[], wgrowtime[])
This also adds data model to the long version information.
This invalidates existing save and bones files due to the changes in
the information at the start of the file.
Some variants were already using a similar approach
using a struct called 'ebones', so adopt the same naming
so NetHack-3.7, hardfought, and some variants are using
the same name.
As before there are fields in the struct that are not
currently used by NetHack-3.7, but the intent is that
hardfought save and bones files can be loaded by
NetHack-3.7 without code modification, for debugging
bug reports.
This invalidates existing save and bones files.
Add a note about NO_TERMS to include/wintty.h for clarity.
Rename tty_startup and tty_shutdown to term_startup() and
term_shutdown(). They are found in termcap.c for !NO_TERMS
like most of the other term_ routines, as well as having
versions for several of the NO_TERMS platforms. They aren't
part of the tty_interface called from the core. The tty
implementation does call and rely on them.
Remove some conditional #ifdef's around term_shutdown()
(formerly tty_shutdown()) and just ensure that all the
tty platforms have an implementation that they can link
with, even if it is just a stub presently.
Put the protype for nethack_exit in extern.h to reduce
maintenance to a single spot, and remove it from other
locations. A warning in the msdos cross-compile led to
this change.
There was a transcription error in the comments in cstd.h for
the standard list of header files, where only the description
remained for <stdlib.h>, not the name of the file itself.
Remove several extraneous inclusions of the standard C99 headers.
Tested on the following afterwards:
Linux (using hints/linux.370) including tty, curses, qt6, and X11
macOS (using hints/macOS.370) including tty, curses, qt5, and X11
Windows MSYS2 using sys/windows/GNUmakefile
Windows Visual Studio using sys/windows/Makefile.nmake
msdos cross-compile on Ubuntu using djgpp cross-compiler
GitHub issue #1315 points out that it is possible for
a downstream function to change an object's nobj field
to point to a completely different chain.
The cited example by @vultur-cadens was:
for (obj = gi.invent; obj; obj = obj->nobj)
if (obj->oclass != COIN_CLASS && !obj->cursed && !rn2(5)) {
curse(obj);
++buc_changed;
}
curse() drops the weapon with drop_uswapwep(),
which calls dropx(),
which calls dropy(),
which calls dropz(),
which calls place_object().
place_object alters the nobj pointer, to point to the floor chain:
otmp->nobj = fobj;
fobj = otmp;
The result was that the next loop iteration was then using floor
objects from the floor chain.
This alters several for-loops to use a more consistent approach,
particularly when the obj is being handed off to a function,
where a downstream function might, or might not, alter the nobj
field.
References:
https://github.com/NetHack/NetHack/issues/1315https://www.reddit.com/r/nethack/comments/1gkc9ub/even_if_you_drop_an_item_before_drinking_from_the/
Changes to setuhpmax() a couple of days ago to deal with sanity_check
for "current hero health as monster better than maximum" ended up
triggering sanity_check about "current hero health better than maximum"
when gaining experience level(s) while polymorphed.
The g? structs had a mix of variables that were written to
the savefile, and those that were not.
For better clarity and to distinguish those that end up in
the savefile, relocate some g? variables that get written
directly to the savefile into different structs.
This updates EDITLEVEL, although technically it probably
didn't need to, since savefile contents are not changing.
Details:
gb.bases -> svb.bases
gb.bbubbles -> svb.bbubbles
gb.branches -> svb.branches
gc.context -> svc.context
gd.disco -> svd.disco
gd.dndest -> svd.dndest
gd.doors -> svd.doors
gd.doors_alloc -> svd.doors_alloc
gd.dungeon_topology -> svd.dungeon_topology
gd.dungeons -> svd.dungeons
ge.exclusion_zones -> sve.exclusion_zones
gh.hackpid -> svh.hackpid
gi.inv_pos -> svi.inv_pos
gk.killer -> svk.killer
gl.lastseentyp -> svl.lastseentyp
gl.level -> svl.level
gl.level_info -> svl.level_info
gm.mapseenchn -> svm.mapseenchn
gm.moves -> svm.moves
gm.mvitals -> svm.mvitals
gn.n_dgns -> svn.n_dgns
gn.n_regions -> svn.n_regions
gn.nroom -> svn.nroom
go.oracle_cnt -> svo.oracle_cnt
gp.pl_character -> svp.pl_character
gp.pl_fruit -> svp.pl_fruit
gp.plname -> svp.plname
gp.program_state -> svp.program_state
gq.quest_status -> svq.quest_status
gr.rooms -> svr.rooms
gs.sp_levchn -> svs.sp_levchn
gs.spl_book -> svs.spl_book
gt.timer_id -> svt.timer_id
gt.tune -> svt.tune
gu.updest -> svu.updest
gx.xmax -> svx.xmax
gx.xmin -> svx.xmin
gy.ymax -> svy.ymax
gy.ymin -> svy.ymin
Related note:
There are some pointer variables that are heads of chains that were not
moved from 'g?' to 'sv?', because they are not actually written to the
savefile directly, but the objects/monst/trap/lightsource/timer in the
chains they point to are. That can be changed, if desired.
Examples: gi.invent, gm.migrating_objs, gb.billobjs, gm.migrating_mons,
gf.ftrap, gl.light_base, gt.timer_base
The old nowrap_add() had different semantics from its replacement,
performing an assignment as well as an addition. Now just does
the addition; its caller needs to perform the assignment.
Yahoo!'s mailer delivered the report about nowrap_add() to my spam
folder, apparently because it thinks that the signature attachments
"may contain harmful content". :-(
nowrap_add() checks for signed overflow after the fact, so after
undefined behavior if that happens.
This rewrites nowrap_add() and moves it from end.c to integer.h.
I haven't generated any values big enough to exercise it, but the
algorithm is straightforward so I'll take it on faith.
Even 100 deaths wasn't enough, as the fuzzer was still quitting.
If we're in wizmode, just execute wizmakemap to recreate the level,
getting rid of whatever was blocking hero from teleporting to safety.
new .h files: hacklib.h selvar.h stairs.h
new .c files: calendar.c, getpos.c, report.c, selvar.c, stairs.c,
strutil.c, wizcmds.c
cleanup of hacklib.c and mdlib.c
hacklib contains functions that do not have to link with the core
relocate wiz commands from cmd.c to wizcmds.c
relocate CRASHREPORT stuff to report.c
relocate getpos stuff from do_name.c to getpos.c
remove temporary struct definition from extern.h
cross-compile PRE-section split into cross-pre1.370 and cross-pre2.370
Windows sys/windows/Makefile.nmake and sys/windows/Makefile.mingw32 and
visual studio project file updates
Unix sys/unix/Makefile.src, sys/unix/Makefile.utl
populate selvar.c and selvar.h
build on MS-DOS (not cross-compile) Makefile updates
for sys/msdos/Makefile.GCC (untested)
vms updates for above (untested)
Replace tabs, split 'if (condtion) do_something' across two lines,
insert lots of spaces in things like 'if(condition){'. I changed
a lot of C++ style comments to traditional C style, but left quite
a few of those as-is.
This also rewrites the code that pull request #1216 purports to fix.
I still can't make sense of the original and the patched edition.
Supersedes #1216Closes#1216
DUMPLOG requests the DUMPLOG feature as it does now
DUMPLOG_CORE requests the internal buffering only (used for CRASHREPORT)
This allows CRASHREPORT to access recent messages without performing
any file I/O.
end.c:177:16: warning: unused parameter 'why' [-Wunused-parameter]
177 | NH_abort(char *why)
| ~~~~~~^~~
end.c: At top level:
end.c:1191:1: warning: 'get_saved_pline' defined but not used [-Wunused-function]
1191 | get_saved_pline(int lineno){
| ^~~~~~~~~~~~~~~
options.c:1263:1: warning: 'optfn_crash_urlmax' defined but not used [-Wunused-function]
1263 | optfn_crash_urlmax(int optidx UNUSED, int req, boolean negated UNUSED, char *opts, char *op)
| ^~~~~~~~~~~~~~~~~~
options.c:1240:1: warning: 'optfn_crash_name' defined but not used [-Wunused-function]
1240 | optfn_crash_name(int optidx UNUSED, int req, boolean negated UNUSED, char *opts, char *op)
| ^~~~~~~~~~~~~~~~
options.c:1217:1: warning: 'optfn_crash_email' defined but not used [-Wunused-function]
1217 | optfn_crash_email(int optidx UNUSED, int req, boolean negated UNUSED, char *opts, char *op)
| ^~~~~~~~~~~~~~~~~
add CRASHREPORT for Windows
add ^P info to report (via DUMPLOG)
new options: crash_email, crash_name, crash_urlmax
new game command: #bugreport
new config option: CRASHREPORT_EXEC_NOSTDERR
new command line option: --bidshow
deleted helper scripts:
NetHackCrashReport.Javascript
nhcrashreport.lua
misc:
update CRASHREPORTURL (will need to be updated before release)
update bitrot in winchain
winchain for Windows
add missing synch_wait for NetHackW --showpaths
add PANICTRACE (and CRASHREPORT) in mdlib.c:build_opts
missing:
packaging (Windows needs the pdb file)
no testing with MSVC command line build
port status:
linux: working, but glibc's backtrace doesn't show static functions
Windows VS: working. pdb file is large - looking into options
MacOS: working
msdos: not supported
VMS: not supported
MSVC: planned, but not attempted
MSYS2: working, but libbacktrace not showing symbols (yet?)
Some functions are passed an obj or monst chain,
and the callers typically don't check them
against 0, so mark them explicitly as NO_NONNULLS
(NO_NONNULLS expands to nothing, but it flags that
some null arg analysis has been done)
Thinko in the reformatting, not triggered during testing because I was
using !CRASHREPORT. With it enabled, setting a watchpoint in gdb on
something unrelated resulted in nethack going into crashreport_init()
and never caming out. (I imagine that it would have eventually but 10
or 15 minutes with nothing noticeable taking place was unbearable, and
the program hadn't even really started yet.)
Fixup some of the inconsistently formatted code that has been
introduced recently or been building up for a while. Done manually.
I wasn't systematic except for looking for lines ending in '&' or '|'
(which wouldn't find such things if they're followed by a comment)
so there might be lots more. I changed a bunch of C++-style //...
comments to old style C /*...*/ so that they'll match the rest of
the core's code rather than because they shouldn't be used.
