This is the third of a series of savefile-related changes.
This adds early-days experimental support for a completely optional
'sfctool' utility (savefile conversion tool), to be able to export
a savefile's contents into a more portable format. There are likely
to be bugs at this stage. In this initial first-attempt, the export
format is a very simple ascii output.
NetHack can be built entirely, without also building this tool.
NetHack has no dependencies on the tool.
Attempts were made to minimize duplication of existing NetHack code.
To achieve that, unfortunately, #ifdef SFCTOOL and #ifndef SFCTOOL
had to be sprinkled around through some of the existing NetHack
source code, so that it could be re-used for building the utility.
The process for building the sfctool typically recompiles the source
files with #define SFCTOOL and a distinct object file with SF- is
produced.
sfctool notes:
Universal ctags is used and required to produce the sfctool utility.
Some targets were added to the Unix and Windows Makefiles to
facilitate the build process.
make sfctool
That should build a copy in util.
Note: At present, the Unix Makefiles do not copy sfctool over to the
NetHack playground during 'make install' or 'make update'.
Until that gets resolved by someone, The tool will
have to be manually copied there by the builder/admin if
desired.
cp util/sfctool ~/nh/install/games/lib/nethackdir/sfctool
Also, a separate Visual Studio sfctool.sln solution was written and
placed in sys/windows/vs. That has has only very limited testing.
Usage:
i) To convert an existing savefile to an exportascii format
that co-resides with the savefile:
sfctool -c savefile
That *must* be executed on the same platform / architecture /
data model that produced the save file in the first place.
ii) To unconvert an existing exportascii format export file to a
historical format savefile that can then be used by NetHack:
sfctool -u savefile
That must be executed on the same target platform / architecture /
data model that was used to build the NetHack that will
utilize the save file that results.
A Windows example:
sfctool -c Fred.NetHack-saved-game
That should result in creation of Fred.NetHack-saved-game.exportascii
from existing savefile:
%USERPROFILE%\AppData\Local\NetHack\3.7\Fred.NetHack-saved-game
A Unix example:
sfctool -c 1000wizard
That should result in creation of 1000wizard.exportascii.gz
from existing savefile in the playground save directory:
1000wizard.gz
Current Mechanics:
1. Makefile recipe, or script uses universal ctags to produce
util/sf.tags.
2. util/sftags is built and executed to read util/sf.tags and
generate: include/sfproto.h and src/sfdata.c.
3. util/sfctool is built from the following:
generated file compiled with -DSFCTOOL:
src/sfdata.c -> sfdata.o
existing files compiled with -DSFCTOOL:
util/sfctool.c -> sfctool.o
util/sfexpasc.c -> sfexpasc.o
src/alloc.c -> sf-alloc.o
src/monst.c -> sf-monst.o
src/objects.c -> sf-objects.o
src/sfbase.c -> sfbase.o
src/sfstruct.c -> sfstruct.o
src/nhlua.c -> sf-nhlua.o
util/panic.c -> panic.o
src/date.c -> sf-date.o
src/decl.c -> sf-decl.o
src/artifact.c -> sf-artifact.o
src/dungeon.c -> sf-dungeon.o
src/end.c -> sf-end.o
src/engrave.c -> sf-engrave.o
src/cfgfiles.c -> sf-cfgfiles.o
src/files.c -> sf-files.o
src/light.c -> sf-light.o
src/mdlib.c -> sf-mdlib.o
src/mkmaze.c -> sf-mkmaze.o
src/mkroom.c -> sf-mkroom.o
src/o_init.c -> sf-o_init.o
src/region.c -> sf-region.o
src/restore.c -> sf-restore.o
src/rumors.c -> sf-rumors.o
src/sys.c -> sf-sys.o
src/timeout.c -> sf-timeout.o
src/track.c -> sf-track.o
src/version.c -> sf-version.o
src/worm.c -> sf-worm.o
src/strutil.c -> strutil.o
This is the second of a series of changes related to save/restore.
No EDITLEVEL bump has been included, because although the code
is changed extensively by this, the content of the savefiles have
not been changed.
Push the use of the structlevel bwrite() and mread() function use
out of the core and into sfstruct.c. This is groundwork for upcoming
changes.
In the core, replace the bwrite() and mread() calls with the
use of type-specific savefile output (Sfo) and savefile
input (Sfi) macros. The macros are defined in a new header file
savefile.h, which also contains the prototypes for the sfo_* and
sfi_* functions that the macros ultimately expand to. The functions
themselves are in src/sfbase.c.
On C99, each Sfo or Sfi macro expansion refers directly to the
corresponding type-specific sfo_* or sfi_* function.
If C23 or later is is use, the majority (all but 3 types) of the
macros refer to a single _Generic output routine sfo(nhfp, dt, tag),
and a single _Generic input routine sfi(nhfp, dt, tag), which handles
the dispatch of the type-specific underlying functions. This was
somewhat experimental, but turned out to be practical because the
compiler would gripe if the type for a variable was not included in
the _Generic when passed as an argument, so it could be fixed.
This alters the savefile verication process by having a common set
return values for the related functions such as uptodate(),
check_version(), etc. The new return values return more information
about savefile incompatibilities, beyond failure/sucess. The
additional information will be useful for an upcoming addition.
The expanded return values are:
SF_UPTODATE (0) everything matched and looks good
SF_OUTDATED (1) savefile is outdated
SF_CRITICAL_BYTE_COUNT_MISMATCH (2) critical size count mismatch
SF_DM_IL32LLP64_ON_ILP32LL64 (3) Windows x64 savefile on x86
SF_DM_I32LP64_ON_ILP32LL64 (4) Unix 64 savefile on x86
SF_DM_ILP32LL64_ON_I32LP64 (5) x86 savefile on Unix 64
SF_DM_ILP32LL64_ON_IL32LLP64 (6) x86 savefile on Windows x64
SF_DM_I32LP64_ON_IL32LLP64 (7) Unix 64 savefile on Windows x64
SF_DM_IL32LLP64_ON_I32LP64 (8) Windows x64 savefile on Unix 64
SF_DM_MISMATCH (9) some other mismatch
The callers in the core have been adjusted to deal with the expanded
return values.
Other miscellaneous inclusions:
- go.oracle_loc -> svo.oracle_loc.
- add a bit (1UL << 30) to called SFCTOOL_BIT as groundwork
for changes to follow.
Not sure why my earlier attempt was unsuccessful. This one isn't as
comprehensive but is simpler and better yet, works as intended.
When saving a level or exiting the program, objects can be deleted
directly rather than having to pass though the objs_deleted list.
This is the first of several savefile-related changes to
follow later. This one is groundwork for those later changes.
Remove internal compression schemes (RLECOMP and ZEROCOMP)
and discard the savefile_info struct that was primarily used to
convey which internal compression schemes had been in use.
Relocate some struct definitions into appropriate header files
for use by code to come in later changes.
Remove the two struct size-related fields from version_info and
from the nmakedefs_s. Instead, include a series of bytes near the
beginning of the savefile, representing the size of each
struct or base data type that impacts the historical savefile
content. Those are referred to as the "critical bytes".
(Related note: the "you" struct required two bytes, low and high,
due to its size).
Compare those critical bytes in a savefile against the NetHack
build that is reading the savefile. This allows mismatch detection
early in the savefile-reading process, and a clean exit, rather than
proceeding to read nonsensical values from the file. Include some
feedback on what the first mismatch was when encountering
one.
For arrays stored in the savefile, use loop-logic in the core
to write/read the array elements one at a time, rather than in
a single blob. This will be required for changes to follow later.
(impacts artiexist[], artidisco[], svd.dungeons[], svl.level_info[],
svl.level.locations[][], msrooms[] field of mapseen, svb.bases[],
svb.disco[] objects[], svm.mvitals[], svs.spl_book[], svd.doors[],
go.oracle_loc[], utrack[], wgrowtime[])
This also adds data model to the long version information.
This invalidates existing save and bones files due to the changes in
the information at the start of the file.
Some variants were already using a similar approach
using a struct called 'ebones', so adopt the same naming
so NetHack-3.7, hardfought, and some variants are using
the same name.
As before there are fields in the struct that are not
currently used by NetHack-3.7, but the intent is that
hardfought save and bones files can be loaded by
NetHack-3.7 without code modification, for debugging
bug reports.
This invalidates existing save and bones files.
There are two hardfought code additions that render save and bones files incompatible
with the upstream NetHack-3.7, and that makes testing with hardfought
save and bones files more challenging than it needs to be, when
investigating and troubleshooting bug reports.
Add some unused fields to advance towards achieving save file parity with
hardfought, which is a significant source of play-testing for NetHack-3.7.
1) the elbereth field addition to u_conduct
This adds an unused placeholder field named 'hf_reserved1', at the appropriate
place in u_conduct to achieve struct field parity with the one in use on
hardfought.
2) hardfought adds a field to struct monst:
char former_rank[25]; /* for bones' ghost rank in their former life */
Instead of adding that to every monst, this adds a new mextra struct
named 'former', which currently contains the equivalent 25-character
field called 'rank' which can hold the content that was in the
former_rank[25] field. That way, the field will only be added when it
is needed.
A pull request https://github.com/k21971/NetHack37/pull/2 has been
done on hardfought to do it the same way (untested there as of yet).
Even though NetHack-3.7 does not utilize that information presently,
this will be a further step toward allowing hardfought-generated save
and bones files to be used for troubleshooting, without modification,
on a similar architecture running stock NetHack-3.7 code.
That savefile parity won't be achieved until the after the
hardfought pull-request mentioned above (or equivalent) is merged.
As this change will not be compatible with existing save and bones
files, it will be accompanied with an EDITLEVEL increment.
The location of the call to dobjsfree() in freedynamicdata()
appears to have been non-ideal.
After getting complaints from a leak-sensing tool after
#quit, the source of the leaks was investigated.
The call to dobjsfree() had been placed immediately following
a call to dmonsfree(), and it did clear out the go.objs_deleted
chain at that point.
Further investigation revealed that the following functions
later on in freedynamicdata() were then adding more deleted
objects to the go.objs_deleted chain:
free_current_level();
freeobjchn(gi.invent);
freeobjchn(gm.migrating_objs);
Move the call to dobjsfree() to a location after those listed
above.
The role, race, gender, and alignment string values are 3 letters
preceded by a dash. I was looking at "name-race-role-gend-algn"
and mistakenly treated them as 4 letters preceded by a dash.
Fixing that changes PL_NSIZ_PLUS and there is one item of that size
written into save files, so the fix invalidates existing save files.
Instead of a menu listing
a - hero1
b - hero2
n - New game
q - Quit
show
a - hero1-role1-race1-gend1-algn1
b - hero2-role2-race2-gend2-algn2
n - New game
q - Quit
or
a - - hero1-role1-race1-gend1-algn1
b - X hero2-role2-race2-gend2-algn2
c - D wizard-role3-race3-gend3-algn3
n - New game
q - Quit
when any game in the list wasn't saved during normal play. (Those
are sorted by character name; the playmode is just coincidence.)
The dash for 'normal' doesn't look great but -/X/D are codes used in
entries written to paniclog. The whole playmode prefix doesn't look
particularly good but I suspect that most players relying on restore
via menu won't see it.
It should work when the character name has dashes in it but that
hasn't been properly tested.
The gender and alignment suffices reflect their value at the time of
save rather than at the start of the game. That might be considered
a bug but it was easiest.
Increments EDITLEVEL; existing save and bones files are invalidated.
The 'spot_monster' option (extra feedback for "accessibility") was
causing messages to be delivered during save, and they could end up
triggering unwanted "--More--" interruptions for tty.
I couldn't reproduce it but it was easy to see where it would happen.
This shuts such messages off in two ways. Only one should be needed
but I'm not sure which one it ought to be.
The g? structs had a mix of variables that were written to
the savefile, and those that were not.
For better clarity and to distinguish those that end up in
the savefile, relocate some g? variables that get written
directly to the savefile into different structs.
This updates EDITLEVEL, although technically it probably
didn't need to, since savefile contents are not changing.
Details:
gb.bases -> svb.bases
gb.bbubbles -> svb.bbubbles
gb.branches -> svb.branches
gc.context -> svc.context
gd.disco -> svd.disco
gd.dndest -> svd.dndest
gd.doors -> svd.doors
gd.doors_alloc -> svd.doors_alloc
gd.dungeon_topology -> svd.dungeon_topology
gd.dungeons -> svd.dungeons
ge.exclusion_zones -> sve.exclusion_zones
gh.hackpid -> svh.hackpid
gi.inv_pos -> svi.inv_pos
gk.killer -> svk.killer
gl.lastseentyp -> svl.lastseentyp
gl.level -> svl.level
gl.level_info -> svl.level_info
gm.mapseenchn -> svm.mapseenchn
gm.moves -> svm.moves
gm.mvitals -> svm.mvitals
gn.n_dgns -> svn.n_dgns
gn.n_regions -> svn.n_regions
gn.nroom -> svn.nroom
go.oracle_cnt -> svo.oracle_cnt
gp.pl_character -> svp.pl_character
gp.pl_fruit -> svp.pl_fruit
gp.plname -> svp.plname
gp.program_state -> svp.program_state
gq.quest_status -> svq.quest_status
gr.rooms -> svr.rooms
gs.sp_levchn -> svs.sp_levchn
gs.spl_book -> svs.spl_book
gt.timer_id -> svt.timer_id
gt.tune -> svt.tune
gu.updest -> svu.updest
gx.xmax -> svx.xmax
gx.xmin -> svx.xmin
gy.ymax -> svy.ymax
gy.ymin -> svy.ymin
Related note:
There are some pointer variables that are heads of chains that were not
moved from 'g?' to 'sv?', because they are not actually written to the
savefile directly, but the objects/monst/trap/lightsource/timer in the
chains they point to are. That can be changed, if desired.
Examples: gi.invent, gm.migrating_objs, gb.billobjs, gm.migrating_mons,
gf.ftrap, gl.light_base, gt.timer_base
Make object deletion work similarly to monster deletion:
it's marked for deletion (by setting the where-field to OBJ_DELETED
and moved to specific deleted-objects chain), but they're actually
freed at the beginning of turn.
This may need some more tweaking, especially in places that iterate
over object chains, but fuzzing did not find any obvious problems.
Fix a case of accessing freed memory: a monster breathed at hero,
destroying some items. The code stored the next item in the chain
(a cloak), but a ring of levitation was destroyed, causing hero to
plop down into lava, destroying the cloak. The item destruction
code then tried to access the destroyed cloak object.
Make the code check the object where-field - which will be different
if the object was marked for deletion. Also removed an extra loop
going through the whole object chain looking for the items to
destroy.
move the custom color data into its own field in the glyphmap
and disassociate it from the unicode/utf8 stuff.
move the glyphcache stuff during options processing and parsing
into new file glyphs.c and out of utf8map.c, and make it
general, and not part of ENHANCED_SYMBOLS.
Do the groundwork for allowing glyph color customizations to
work when any symset is loaded and not restrict it only to
the enhanced1 H_UTF8 symsets.
The customizations in effect are still affiliated with a particular
symset.
Also closes#1224, but the PR itself references a data structure
made obsolete by this commit. The curses comment from the PR was
added into the code.
The PR also made several suggestions, but only the first
one has been included in this commit (and no longer based on
the handler), that being:
"allow defining colors if other symbol handling modes are used
(possibly limited to the standard 16 colors)."
FredrIQ also wrote the following suggestions in PR#1224:
Something I was also contemplating, unrelated to implementation of this
support in curses, would be the ability for the following:
allow defining colors if other symbol handling modes are used (possibly limited to the standard 16 colors)
allow defining attributes (for example: glyph:G_pet_female_kitten:U+0066/red/underline)
allow specifying glyphs as wildcards for defining global color/attribute changes
Something I also want to see are keywords for "don't change the current defined data". If this
were to be added, you could for example do this:
OPTIONS=glyph:G_*_fox:U+0064/blue
OPTIONS=glyph:G_statue_*:basechar/gray/underline
for "make all foxes use a blue color, make all statues gray with underline" without needing
to specify the relevant character for every statue. This ("basechar", "basefg", etc)
should perhaps also be added for MENUCOLORS and statushilites, so that you can, for
example, underline all items being worn without needing to specify a bunch of
near-duplicate rules for combining BUC colors + underline worn items
as per #1064
The new change to reset discoveries and monster-stats when exiting
the tutorial used dynamic data which wouldn't be freed if player
used #quit and declined to resume the regular game.
It turns out that such a leak was already present for start-of-game
inventory that gets stashed away during the tutorial.
In both cases, it could happen at most once per game so wasn't a big
deal as far as memory leaks go.
DUMPLOG requests the DUMPLOG feature as it does now
DUMPLOG_CORE requests the internal buffering only (used for CRASHREPORT)
This allows CRASHREPORT to access recent messages without performing
any file I/O.
Re-use the array allocated for iterating over all monsters during
monster movement much of the time. It was being allocated from
scratch for each round of monster movement, then freed after they
moved, then repeated the next round.
While hunting for a memory leak in object allocation--which I haven't
found yet--I discovered one in monster movement. iter_mons_safe()
allocates an array of (monst *) pointers for the monsters on the
current level, loops over that array to call a function for each
one, then frees the array. But if the game ends while that called
function is running, execution never returns to iter_mons_safe() so
it wasn't able to free the memory.
Since that can happen at most once per game, it wasn't a signifcant
leak. This fixes it anyway.
There was a second issue: make sure that iter_mons_safe() doesn't
call alloc(0) to make the temporary array for zero monsters when
there aren't any on the level. That might not be able to happen for
monster movement but the routine is written to be more general than
just movement. alloc(0) could confuse the MONITOR_HEAP code. In
C89/C90 I think malloc(0) is allowed to return NULL (don't recall
for sure; maybe that was just known pre-standard behavior for some
implementations). Null return would trigger a panic even without
MONITOR_HEAP. Don't know about C99 and later.
Some functions are passed an obj or monst chain,
and the callers typically don't check them
against 0, so mark them explicitly as NO_NONNULLS
(NO_NONNULLS expands to nothing, but it flags that
some null arg analysis has been done)
The tracks left by hero were cleared when player saved and
restored the game, or changed levels. Now the tracks are
saved in the dungeon level, so changing levels keeps the tracks
left by hero in that level.
Also increased the length of tracks from 50 to 100, and
simplify the tracking function.
Thing not done: fade out old tracks when returning to a level.
Breaks saves and bones.
Adds a new lua command
des.exclusion({ type = "teleport", region = { x1,y1, x2,y2 } });
which allows defining "exclusion zones" in the level, areas where
random teleports (or falling into the level) will never place the hero.
Does not prevent targeted teleportation into the area.
Breaks saves and bones.
When returning to play from within the tutorial, remove the level files
similar to how they're discarded for the rest of the dungeon when going
into the endgame. It turned out to be a bit messier than anticipated.
The dungeon.c bit is sufficient for #overview, which now hides regular
level 1 while in the tutorial and hides all tutorial levels once exited.
Those will still appear in end-of-game disclosure.
Reported five months ago, a save was performed while a mounted hero
was engulfed. Restore issued a warning about the engulfer being
placed on top of the steed (who shouldn't have been on the map).
The report arrived at about the same time as engulfing a riding
hero was changed to force a dismount instead of engulfing both hero
and steed so nothing further was done about it. This changes
restore to not put a steed on the map and then take it off again.
It also attempts to simplify usteed and ustuck handling during save
and restore.
Testing so far indicates that things are still working correctly.
Keep makeplural(body_part(FINGER)) crossed.
Existing save and bones files are invalidated.
A number of C compiler suites have a math.h library that includes a yn()
function name that conflicts with NetHack's yn() macro:
"The y0(), y1(), and yn() functions are Bessel functions of the second kind,
for orders 0, 1, and n, respectively. The argument x must be positive. The
argument n should be greater than or equal to zero. If n is less than zero,
there will be a negative exponent in the result."
At one point, isaac64.h included math.h, although that has since been removed.
Some libraries used in NetHack (Qt for one) do include math.h and that required
build work-arounds to avoid the conflict.
Rename the NetHack macro from yn() to y_n() and avoid the math.h conflict
altogether, eliminating the need for that particular work-around.
Keep track of how a role|race|gender|alignment option got its value
so that role:!Tourist in .nethackrc and role:!Priest in NETHACKOPTIONS
yield 'role:!Priest' rather than merging into 'role:!Priest !Tourist'.
It also doesn't write the value into new config file for #saveoptions
if that value comes from environment or command line (not applicable
since the command line arguments for role,&c don't go through options
handling). Also, the old config file value takes precedence over
the current game's value file so that 'role:random' doesn't become
'role:Healer' or such in a new config after the random value gets
picked for play.
This only tracks the role, race, gender, and alignment options but the
concept could be extended to all options. The data would need to be
saved and restored if values set interactively need to be retained in
restore sessions (doesn't apply to role,&c since those don't change
during play).
The consolidation of global variables from scattered source
files into decl.c and declared in decl.h was begun in 3.7.0.
Their placement in common files was done for centralized
initialization and potential re-initialization during a
"play again" scenario.
It wasn't really necessary for all of them to be housed in a
single huge structure to meet the "play again" requirement,
and the single huge structure has been a little unwieldy when
it comes to maintenance.
Following this commit, instead of one single extremely large structure
named 'g' to house all of the relocated global variables, they
are distributed into several ga through gz.
To make things easy for the developer, each variable is placed
into the struct corresponding to the starting letter of the variable.
That way, no lookup is required in order to know which struct houses
a particular variable, it is a simple match to the starting letter
for all the centralized global variables.
A global variable named 'amulets', would be found in ga.
ga.amulets
^ ^
A global varable named 'move', would be found in gm.
gm.moves
^ ^
A global variable named 'val_for_n_or_more' would be found in gv.
gv.val_for_n_or_more
^ ^
A global variable named 'youmonst' would be found in gy.
gy.youmonst
^ ^
Trap doors saved their destinations as an absolute level, rather than a
relative one, so if you loaded bones from a special level their
destinations would reflect the dungeon layout from the bones player's
game. For example, die on the Oracle level, on dlvl5, with a trap door
that goes to dlvl6. Another player gets those bones on their Oracle
level, which is dlvl8... the trap door would still go to dlvl6. Pretty
amazing trap door -- something you might see in a funhouse!
Include relative rather than absolute destinations in save and bones
files, much like stairs do, to avoid this problem.
I bumped EDITLEVEL because although this won't break save files in an
obvious way, it will interpret the (absolute) destinations in existing
save and bones files as relative, leading to some crazy long falls. :)
The pull request included some changes that were neither accidental nor
unintentional, so only a subset of the changes from pull request #869
submitted by klorpa were manually applied.
behaviour -> behavior
speach -> speech
knowlege -> knowledge
incrments -> increments
stethscope -> stethoscope
staiway -> stairway
arifact -> artifact
extracing -> extracting
The uses of "iff" were left alone.
Close#869
