From: Not Zed <notzed@gmail.com>
Date: Tue, 4 Jan 2022 22:49:15 +0000 (+1030)
Subject: Pattern matching and defaults for structs
X-Git-Url: https://code.zedzone.au/cvs?a=commitdiff_plain;h=6e4db7e40a15cc8164c2cce949ffaf656f473426;p=panamaz

Pattern matching and defaults for structs
Added ffmpeg example.
Changed test-api to be c-like example.
---

diff --git a/README b/README
index 6fe7c06..cdb34e6 100644
--- a/README
+++ b/README
@@ -2,91 +2,53 @@
 Introduction
 ------------
 
-This is an alternative to jextract for the OpenJDK panama project
-which in turn is a project to implement Java to Native interfaces at
-runtime.
+This is some experiments with the panama-foreign abi JEP for invoking
+native C functions from Java directly and without JNI.
 
-It uses a gcc plugin for type extraction and a "simple" perl script to
-transform this into Java source files with the appropriate annotations
-for panama to do it's thing.
+The main goal is to experiment with creating a "java friendly" and
+mostly type-safe api directly in one step, without requiring
+additional wrapping or inconvenient use.
 
-Compile
--------
-
-Edit the makefile to point to your gcc directory and make.
-
-$ cd src
-$ vi Makefile
-$ make
-
-generate is a perl script.
-
-See test-api/* for a simple example.  JAVA_HOME must point to
-a compatible panama-enabled jdk.
-
-Use dump.so
------------
-
-dump.so is a gcc plugin which captures structures, unions, enums,
-exported functions, and a few typedefs into a perl source file
-containing a single hash.
-
-Put all the include files required for your api in api.h.
-
-$ gcc -fplugin=src/export.so -fplugin-arg-export-output=api.pm api.h -o /dev/null
+It's roughly tracking the openjdk repository of the same date.
 
-api.pm is a text file of perl source-code which contains a descriptor
-of all supported types found.
+It uses a gcc plugin to compile the headers to obtain most of the api
+information, but requires cpp and perl to extract the values of
+#define constants.
 
-Use generate
-------------
+This api information is then converted to Java wrappers using a
+config-directed perl script `generate-native'.  The config is flexible
+enough to either generate c-like libraries of static functions or
+object-oriented layouts with data, static and object functions grouped
+by class.
 
-Quick and dirty:
+test-vulkan uses a different approach as the vulkan api is officially
+defined by an xml registry and not the generated C headers.  This is
+directly converted to about-as-java-friendly a vulkan api as one can
+hope for (all those structs you need to initialise get messy in any
+language).  Most of the script is converting the registry vk.xml file
+into a usable structure as it's really only designed for outputting c
+headers.
 
-$ generate -d dir -t name.space -c APILib -lapi ./api.pm
+Compile
+-------
 
-Generate all structures in api.pm (the ./ is required) and place all
-functions into dir/name/space/APILib.java which will link with
-libapi.so.
+Requirements are gcc and cpp, perl, GNU make.
 
-The generator is quite flexible, sub-sets of functions can be placed
-in alternative *Lib.java files, types can be filtered.  If a function
-or type is specified explicitly, then only those types it requires are
-included in the generated output (recursively).
+First run make in src.
 
-See the top of the file for details, it's probably up to date.
+See test-*/* for examples.  JAVA_HOME must point to a compatible
+panama-enabled jdk.
 
 Status
 ------
 
-This is currently just a 'quick and dirty' weekend hack.  The tools
-spew a lot of debug.  dump.cc is basic c-compatible-c++, because I
-don't know c++.
-
-dump.so works by hooking into the gcc pre-compiled headers function,
-it might work with a c source file as well.  It's only been used with
-gcc 9.
-
-The dumper will hard-fail if it sees defintions it doesn't understand,
-and currently it's only been tested with a small nubmer of libraries.
-
-Enumerations are currently translated to integers.  The generator
-doesn't currently export any of the symbolic values to Java but it is
-easy to add.
-
-varags is not implemented.
-
-All function pointers are translated to a unique set of functional
-interfaces whose names are based on the function signature.  So the
-type names are ugly but they are stable.
-
-Anonymous type names are based on the structure member they belong to
-so they may not be unique.
-
-It will only ever support c.
+It's all work in progress of course.
 
-Only tested against the panama 'openjdk-14-panama+1-15_linux' on
-slackware-current @ 2020.01.01 and gcc 9.2.0.
+* bitfields not implemented yet.
+* varargs is not implemented.
+* the support library in Memory.java is copied to each output api, but should
+  be global.
+* the scope and object lifecycle stuff is not really sorted out yet.
 
 License
 -------
@@ -99,3 +61,4 @@ Links
 
  * https://www.zedzone/software/panamaz.html - project page.
  * https://openjdk.java.net/projects/panama - openjdk panama page.
+ * https://github.com/openjdk/panama-foreign - openjdk panama source.
diff --git a/src/export-defines b/src/export-defines
index ece2b0e..79b35e5 100755
--- a/src/export-defines
+++ b/src/export-defines
@@ -1,91 +1,345 @@
 #!/usr/bin/perl
 
-# the gcc plugin doesn't have access to the #defines, so this is a 'simple' way to get them.
+use Data::Dumper;
 
-@matchDefine = ();
-@matchInclude = ();
-
-@includes = ();
+my @includes = ();
+my $header;
+my $control = "api-defines.def";
+my $output;
 
 while (@ARGV) {
-    my $cmd = shift(@ARGV);
-
-    if ($cmd eq "-d") {
-	my $v = shift(@ARGV);
-	push @matchDefine, $v;
-    } elsif ($cmd eq "--define-file") {
-	my $file = shift(@ARGV);
-	push @matchDefine, readMatchFile($file);
-    } elsif ($cmd eq "-i") {
-	my $v = shift(@ARGV);
-	push @matchInclude, $v;
-    } elsif ($cmd eq "-I") {
-	my $v = shift(@ARGV);
-	push @includes, $v;
-    } elsif ($cmd =~ "-I(.*)") {
-	push @includes, $1;
-    } else {
-	$header = $cmd;
-    }
+	my $cmd = shift;
+
+	if ($cmd eq "-t") {
+		$package = shift;
+    } elsif ($cmd eq "-d") {
+		$output = shift;
+    } elsif ($cmd eq "-v") {
+		$verbose++;
+	} elsif ($cmd eq "-I") {
+		push @includes, shift;
+	} else {
+		$control = $cmd;
+	}
+}
+
+die ("no output specified") if !$output;
+
+my $defs = loadControlFile($control);
+my @exports = @{$defs->{define}};
+my %rawDefines = (); # indexed by header
+
+my $CPPFLAGS = "";
+
+foreach $i (@includes) {
+	$CPPFLAGS .= " '-I$i'";
+}
+
+# flatten the generic format to a more usable format, work out defaults (. and syntax check?)
+foreach $export (@exports) {
+	my $includes = 0, $excludes = 0;
+
+	foreach $inc (@{$export->{items}}) {
+		my @options = @{$inc->{options}};
+
+		if ($inc->{match} =~ m@^/(.*)/$@) {
+			$inc->{regex} = qr/$1/;
+		} else {
+			$inc->{regex} = qr/^$inc->{match}$/;
+		}
+
+		$inc->{mode} = "include";
+		foreach $o (@{$inc->{options}}) {
+			if ($o =~ m/^(exclude|include|file-include|file-exclude)$/) {
+				$inc->{mode} = $o;
+			}
+		}
+		if ($inc->{mode} =~ m/include/) {
+			$includes += 1;
+		} else {
+			$excludes += 1;
+		}
+	}
+
+	$export->{default} = 'all';
+	$export->{default} = 'none' if ($includes > 0);
+
+	my @options = @{$export->{options}};
+
+	foreach $o (@options) {
+		if ($o =~ m/header=(.*)/) {
+			$export->{header} = $1;
+		} elsif ($o =~ m/default=(.*)/) {
+			$export->{default} = $1;
+		} elsif ($#options == 0) {
+			$export->{header} = $o;
+		} else {
+			print STDERR "unknown defines option '$o'\n";
+		}
+	}
+
+	# insert ignore <built-in> thing
+	if ($export->{default} eq 'all') {
+		unshift @{$export->{items}}, { regex => qr/<built-in>/, mode => 'file-exclude' };
+	}
+
+	die ("no header for '$export->{name}'") if !$export->{header};
+}
+
+# load all defines once and link in
+# keep_comments is a bit broken
+foreach $export (@exports) {
+	my $header = $export->{header};
+
+	if (!defined($rawDefines{$header})) {
+		$rawDefines{$header} = scanDefines($header, { CPPFLAGS=>$cppflags, keep_comments=>0 });
+	}
+
+	$export->{rawDefines} = $rawDefines{$header};
+}
+
+foreach $export (@exports) {
+	# extract matching #defines
+	my @defines = ();
+
+	foreach $d (@{$export->{rawDefines}}) {
+		my $output = 0;
+		my $matches = 0;
+
+		print "? $d->{name} $d->{file}" if $verbose;
+		foreach $inc (@{$export->{items}}) {
+			if ($inc->{mode} eq "include") {
+				$matches = $d->{name} =~ m/$inc->{regex}/;
+				$output = 1 if $matches;
+			} elsif ($inc->{mode} eq "exclude") {
+				$matches = $d->{name} =~ m/$inc->{regex}/;
+			} elsif ($inc->{mode} eq "file-include") {
+				$matches = $d->{file} =~ m/$inc->{regex}/;
+				$output = 1 if $matches;
+			} elsif ($inc->{mode} eq "file-exclude") {
+				$matches = $d->{file} =~ m/$inc->{regex}/;
+			}
+			print " ($inc->{mode} '$inc->{match}' =$matches)" if $verbose;
+			last if $matches;
+		}
+
+		$output = 1 if (!$matches && $export->{default} eq "all");
+
+		print " output=$output\n" if $verbose;
+
+		push (@{$export->{export}}, $d) if $output;
+	}
+
+}
+
+open (my $fh, ">", "$output~") || die("can't open $output~ for writing");
+
+export($fh, \@exports);
+
+close ($fh) || die("error writing");
+rename "$output~",$output || die("error overwriting $output");
+
+exit 0;
+
+# ######################################################################
+
+sub uniq {
+  my %seen;
+  return grep { !$seen{$_}++ } @_;
+}
+
+sub export {
+	my $fp = shift;
+	my @exports = @{shift @_};
+
+	print $fp <<END;
+#include <stdio.h>
+#include <stdint.h>
+END
+	foreach $h (uniq map { $_->{header} } @exports) {
+		print $fp  "#include \"$h\"\n";
+	}
+
+	# this is effectively a function overloading system so that the
+	# compiler can select the datatype of the evaluation of the
+	# definition.
+	print $fp <<END;
+/* note unsigned types are output as signed for java compatability */
+#define FMT(x) \\
+	__builtin_choose_expr(__builtin_types_compatible_p(typeof(x), float),    "value=>%a, type=>'f32'", \\
+	__builtin_choose_expr(__builtin_types_compatible_p(typeof(x), double),   "value=>%a, type=>'f64'", \\
+	__builtin_choose_expr(__builtin_types_compatible_p(typeof(x), int32_t),  "value=>%d, type=>'i32'", \\
+	__builtin_choose_expr(__builtin_types_compatible_p(typeof(x), uint32_t), "value=>%d, type=>'u32'", \\
+	__builtin_choose_expr(__builtin_types_compatible_p(typeof(x), int64_t),  "value=>%ld, type=>'i64'", \\
+	__builtin_choose_expr(__builtin_types_compatible_p(typeof(x), uint64_t), "value=>%ld, type=>'u64'", \\
+	__builtin_choose_expr(__builtin_types_compatible_p(typeof(x), typeof(char[])),  "value=>'%s', type=>'string'", "type => undefined" )))))))
+
+int main(int argc, char **argv) {
+ FILE *fp = fopen(argv[1], "w");
+
+ fputs("{\\n", fp);
+END
+
+	foreach $export (@exports) {
+		print $fp <<END;
+fputs("'define:$export->{name}' => { name => '$export->{name}', type => 'define', values => [\\n", fp);
+END
+		foreach $d (@{$export->{export}}) {
+			my $docomment;
+
+			if ($d->{comment}) {
+				my $comment = $d->{comment};
+				$comment =~ s@(['"])@\\\\\\$1@g;
+				$comment =~ s@\n@\\n@g;
+				$docomment = ", comment => '$comment' ";
+			}
+
+			print $fp <<END;
+ fputs("  { name => \\"$d->{name}\\", ", fp);
+ fprintf(fp, FMT($d->{name}), ($d->{name}));
+ fputs("$docomment},\\n", fp);
+END
+		}
+		print $fp <<END;
+ fputs(" ],\\n},\\n", fp);
+END
+	}
+	print $fp <<END;
+ fprintf(fp, "}\\n");
+ fclose(fp);
+}
+END
 }
 
-my $all = join ('|', @matchDefine);
+# args: filename, \@defines, \@defineList
+# export a c file generator
+sub export_generator {
+	my $filename = shift;
+	my @defines = @{shift @_};
+	my @defineList = @{shift @_};
+}
 
-$matchDefine = qr/($all)/;
+# args: header, \%options
+# options = {
+#   CPPFLAGS => 'flags'
+#   keep_comments => 1 to keep comments
+# }
+# returns a list of
+# {
+#   name=>'name',
+#   comment='comment',
+#   file='filename',
+#   line=linenumber
+# }
+sub scanDefines {
+	my $header = shift;
+	my %o = %{shift @_};
+    my $lastc = "";
+	my $source;
+	my $sourceLine;
 
-$all = join('|', @matchInclude);
+    print STDERR "Scanning $header\n";
 
-$matchInclude = qr/($all)/;
+    open (my $in,"-|","cpp -dD ".($o{keep_comments} ? '-CC' : '')." $o{CPPFLAGS} $header") || die("Can't find include file: $header");
+    while (<$in>) {
+		# line markers
+		if (m/^\# (\d+) \"([^\"]*)/) {
+			$sourceLine = $1;
+			$source = $2;
+			next;
+		}
+		# accumulate comments
+		# single line comments override multi-line
+		if ($o{keep_comments}) {
+			if (m@/\*(.*)\*/@) {
+				do {
+					$lastc = $1 if $lastc eq "";
+					s@/\*.*\*/@@;
+				} while (m@/\*(.*)\*/@);
+			} elsif (m@/\*(.*)@) {
+				my $com = "$1\n" if $1;
+				while (<$in>) {
+					chop;
+					if (m@(.*)\*/@) {
+						$com .= $1 if $1;
+						last;
+					} else {
+						$com .= "$_\n";
+					}
+				}
+				$lastc = $com if $com && $lastc eq "";
+			} elsif (m@//(.*)@) {
+				$lastc = $1 if $1;
+				s@//.*@@;
+			}
+		}
 
-print "all: $all\n";
-print "$matchDefine\n";
+		if (m/^\s*#define\s+(\w*)\(/) {
+			# ignore macros
+			$lastc = "";
+		} elsif (m/^\s*#define\s+(\w*)\s*$/) {
+			# ignore empty defines
+			$lastc = "";
+		} elsif (m/^\s*#define\s+(\w+)/) {
+			my $name = $1;
+			my %define = ();
 
-sub readMatchFile {
-    my $path = shift @_;
-    my @lines = ();
+			$define{name} = $name;
+			$define{comment} = $lastc if $lastc ne "";
+			$define{file} = $source;
+			$define{line} = $sourceLine;
 
-    open(my $f,"<$path");
-    while (<$f>) {
-	chop;
-	next if m/^#/;
+			push @defines, \%define;
 
-	push @lines, $_;
-    }
-    close($f);
+			$lastc = "";
+		}
 
-    my $all = join ('|', @lines);
+		$sourceLine++;
+	}
+	close $in;
 
-    return qr/^($all)$/;
+	return \@defines;
 }
 
-$dump = 0;
-
-$args = join("", map { "\"-I$_\"" } @includes);
-
-open IN,"gcc -C -E -dD $args $header|";
-# only #defines that are defined in the same directory or file as header?
-while (<IN>) {
-    if (m/^# (\d*) \"(.*)\"/) {
-	$dump = ($2 =~ /$matchInclude/);
-    }
-    if ($dump) {
-	if (m/^#define\s+(\w*)\s+(.*)/) {
-	    my $name = $1;
-	    my $def = $2;
-
-	    if ($name =~ /$matchDefine/) {
-		if ($def =~ m/L$/) {
-		    print "public static final long $name = (long)$def;\n"; if ($def =~ m/L$/);
-		} elsif ($def =~ m/^(0x)?[0-9]*$/) {
-		    print "public static final long $name = (int)$def;\n"; if ($def =~ m/L$/);
-		} elsif ($def =~ m/f$/) {
-		    print "public static final float $name = $def;\n"; if ($def =~ m/L$/);
-		} elsif ($def =~ m/^[\.E-+0-9]*$/) {
+# TODO: library
+sub loadControlFile {
+	my $path = shift @_;
+	my %def = ();
+	my $target;
+
+	open (my $d,"<",$path);
+
+	while (<$d>) {
+		next if /\s*\#/;
+
+		chop;
+
+		if ($target) {
+			if (m/\s*\}\s*$/) {
+				undef $target;
+			} elsif (/^\s*(\S+)\s*(.*)/) {
+				my @options = split(/\s+/,$2);
+				push @{$target->{items}}, {
+					match => $1,
+					options => \@options
+				};
+			}
+		} elsif (/^(\w+)\s+(\S*)\s*(.*)\s+\{/) {
+			my @options = split(/\s+/,$3);
+
+			$target = {
+				type => $1,
+				name => $2,
+				options => \@options,
+				items => []
+			};
+			push @{$def{$1}}, $target;
+		} elsif (/\S/) {
+			die("invalid line: %_");
 		}
-		#if ($def =~ m/^0x/) {
-		#}
-	    }
 	}
-    }
+
+	close $d;
+
+	return \%def;
 }
-close IN;
diff --git a/src/export.cc b/src/export.cc
index 4a34c58..a37f0bd 100644
--- a/src/export.cc
+++ b/src/export.cc
@@ -697,7 +697,7 @@ static void export_type(tree type, const char *names) {
 				names, names, size, TYPE_UNSIGNED(target) ? 'u' : 'i', size);
 
 			for (tree v = TYPE_VALUES(target); v != NULL; v = TREE_CHAIN (v)) {
-				generate("\t{ label => '%s', value => '%ld' },\n",
+				generate("\t{ name => '%s', value => '%ld' },\n",
 					IDENTIFIER_POINTER(TREE_PURPOSE(v)),
 					tree_to_shwi(TREE_VALUE(v)));
 			}
@@ -873,7 +873,7 @@ static void export_type(tree type, const char *names) {
 			names, names, size, TYPE_UNSIGNED(type) ? 'u' : 'i', size);
 
 		for (tree v = TYPE_VALUES(type); v != NULL; v = TREE_CHAIN (v)) {
-			generate("\t{ label => '%s', value => '%ld' },\n",
+			generate("\t{ name => '%s', value => '%ld' },\n",
 				IDENTIFIER_POINTER(TREE_PURPOSE(v)),
 				tree_to_shwi(TREE_VALUE(v)));
 		}
@@ -977,7 +977,7 @@ static void plugin_finish(void *event_data, void *user_data) {
 	for (struct node *n = dumped.list.head; n; n=n->next)
 		generate("# %s\n", n->name);
 
-	generate(");\n");
+	generate("}\n");
 	fclose(output_file);
 
 	if (debug_level > 0)
@@ -1007,7 +1007,7 @@ int plugin_init(struct plugin_name_args *plugin_info, struct plugin_gcc_version
 		exit(EXIT_FAILURE);
 	}
 
-	generate("%%data = (\n");
+	generate("{\n");
 
 	register_callback(plugin_info->base_name, PLUGIN_FINISH_DECL, plugin_finish_decl, NULL);
 	register_callback(plugin_info->base_name, PLUGIN_FINISH_TYPE, plugin_finish_type, NULL);
diff --git a/src/generate-native b/src/generate-native
index 55eaee7..62cf2fb 100755
--- a/src/generate-native
+++ b/src/generate-native
@@ -2,8 +2,14 @@
 
 # -*- Mode:perl; perl-indent-level:4;tab-width:4; -*-
 
-# TODO: get/set should take a typed field for struct/union *
-# TODO: a flag for func_name to funcName option
+# TODO: global defaults and/or pattern matching on api targets, partially done
+# TODO: perhaps an option which just dumps everything it finds at some level
+# TODO: the code here isn't a complete mess but could be improved
+#       - parameterise some functions
+# TODO: map int to boolean where appropriate
+# TODO: arrays with specified lengths passed as arguments could be size-checked in function stubs.
+# TODO: error codes -> exceptions?
+# TODO: auto-loading of libraries (library xx load=blah) option.
 
 use Data::Dumper;
 use File::Basename;
@@ -14,11 +20,13 @@ $scriptPath = dirname(__FILE__);
 $package = "";
 $output = "bin";
 $verbose = 0;
+$apidef = "api.def";
 
 # usage
 #  -t package    target package
 #  -d directory  output root
 #  -v            verbose
+#  -a datafile   add datafile to the dataset, can be from export.so or export-defines, etc
 
 my %typeSizes = (
 	i8 => 'byte', u8 => 'byte',
@@ -29,6 +37,26 @@ my %typeSizes = (
 	f64 => 'double',
 	);
 
+# or just use some formatting function table
+my %defineType = (
+	%typeSizes,
+	string => 'String'
+	);
+
+my %definePrefix = (
+	i8 => '(byte)',
+	u8 => '(byte)',
+	i16 => '(short)',
+	u16 => '(short)',
+	string => '"'
+	);
+my %defineSuffix = (
+	u64 => 'L',
+	i64 => 'L',
+	f32 => 'f',
+	string => '"'
+	);
+
 my %intSizes = ( 8 => 'byte', 16 => 'short', 32 => 'int', 64 => 'long' );
 my %typePrimitive = (
 	"byte" => 8,
@@ -50,28 +78,337 @@ my %typeSignature = (
 	"MemoryAddress" => "Ljdk/incubator/foreign/MemoryAddress;",
 	);
 
+my %renameTable = (
+	'studly-caps' => sub { my $s = shift; $s =~ s/(?:^|_)(.)/\U$1/g; return $s; },
+	'camel-case' => sub { my $s = shift; $s =~ s/(?:_)(.)/\U$1/g; return $s; },
+	'upper-leadin' => sub {	my $s = shift; $s =~ s/^([^_]+)/\U$1/; return $s; },
+	'identity' => sub {	return $_[0]; },
+	);
+
+my %data = ();
+
 while (@ARGV) {
-	my $cmd = shift(@ARGV);
+	my $cmd = shift;
 
 	if ($cmd eq "-t") {
-		$package = shift(@ARGV);
+		$package = shift;
     } elsif ($cmd eq "-d") {
-		$output = shift(@ARGV);
+		$output = shift;
     } elsif ($cmd eq "-v") {
 		$verbose++;
+    } elsif ($cmd eq "-a") {
+		my $file = shift;
+		my $info;
+
+		print "Add $file\n";
+
+		unless ($info = do $file) {
+			die "couldn't parse $file: $@" if $@;
+			die "couldn't import $file: $!"    unless defined $info;
+			die "couldn't run $file"       unless $info;
+		}
+
+		%data = (%data, %{$info});
 	} else {
-		$meta = $cmd;
+		$apidef = $cmd;
 	}
 }
 
-# load in interface file
-do $meta;
+my $api = loadControlFile($apidef);
 
+analyseAPI($api);
 analyseAndFixTypes();
 
-if ($verbose) {
+if (0) {
+	$s = $data{'struct:AVCodecContext'};
+	$obj = findAPIObject($api, 'struct', $s->{name});
+	print Dumper($obj);
+
+	foreach $m (@{$s->{fields}}) {
+		$inc = findAPIField($obj, $m->{name});
+		print " $m->{name} - $inc\n";
+	}
+}
+
+my $toDump = analyseDependencies(\%data, findRoots($api));
+
+if ($verbose > 1) {
 	print "Using:n";
 	print Dumper(\%data);
+	print "API:n";
+	print Dumper($api);
+}
+
+# find api.struct that matches a given struct name
+sub findAPIObject {
+	my $api = shift;
+	my $type = shift;
+	my $name = shift;
+
+	print "find api for $type:$name\n" if $verbose;
+	foreach $obj ( @{$api->{$type}} ) {
+		next if $obj->{name} eq '<default>';
+		print " $obj->{name} ? $name\n" if $verbose;
+		if ($obj->{name} =~ m@/(.*)/@) {
+			my $rx = qr/$1/;
+			return $obj if ($name =~ m/$rx/);
+		} elsif ($obj->{name} eq $name) {
+			return $obj;
+		}
+	}
+
+	print " -> fallback=$type:<default>\n" if $verbose && $api->{"$type:<default>"};
+	return $api->{"$type:<default>"};
+}
+
+# sub findAPIStruct {
+# 	my $api = shift;
+# 	my $name = shift;
+
+# 	foreach $obj ( @{$api->{struct}} ) {
+# 		next if $obj->{name} eq '<default>';
+# 		print "$obj->{name} ? $name\n" if $verbose;
+# 		if ($obj->{name} =~ m@/(.*)/@) {
+# 			my $rx = qr/$1/;
+# 			return $obj if ($name =~ m/$rx/);
+# 		} elsif ($obj->{name} eq $name) {
+# 			return $obj;
+# 		}
+# 	}
+
+# 	return $api->{'struct:<default>'};
+# }
+
+sub findAPIField {
+	my $obj = shift;
+	my $name = shift;
+
+	foreach $inc (grep { $_->{mode} eq 'field' } @{$obj->{items}}) {
+		return $inc if $name =~ m/$inc->{regex}/;
+	}
+}
+
+# find all directly referenced types and field types from api
+sub findRoots {
+	my $api = shift;
+	my %seen;
+
+	foreach $obj ( @{$api->{library}} ) {
+		foreach $inc ( @{$obj->{items}} ) {
+			if ($inc->{mode} eq 'func') {
+				my @list = grep { $_->{type} eq $inc->{mode} && $_->{name} =~ m/$inc->{regex}/ } values %data;
+
+				foreach $func (@list) {
+					$seen{"func:$func->{name}"} ++;
+				}
+			}
+		}
+	}
+
+	# all defines included
+	foreach $def ( @{$api->{define}} ) {
+		$seen{"define:$def->{name}"} ++;
+	}
+
+	foreach $obj ( @{$api->{struct}}, @{$api->{func}}) {
+		my @list;
+
+		if ($obj->{name} =~ m@/(.*)/@) {
+			my $rx = "$obj->{type}:$1";
+			push @list, grep { $_ =~ m/$rx/ } keys %data;
+		} else {
+			push @list, "$obj->{type}:$obj->{name}";
+		}
+
+		foreach $n (@list) {
+			$seen{$n} ++;
+		}
+
+	}
+
+	delete $seen{'struct:<default>'};
+	delete $seen{'func:<default>'};
+	delete $seen{'call:<default>'};
+
+	my @list = sort keys %seen;
+
+	return \@list;
+}
+
+# analyse dependencies of the supplied roots
+# only fields actually referenced in the api.def file are included
+# \%seen = \%data, \@roots
+sub analyseDependencies {
+	my $data = shift;
+	my @roots = @{shift @_};
+	my %seen;
+
+	print "Finding dependencies of $#roots roots\n";
+
+	while ($#roots >= 0) {
+		my $name = shift @roots;
+		my $s = $data{$name};
+		my @list;
+
+		next if $seen{$name}++;
+
+		print "visit $name $s->{name}\n" if $verbose;
+
+		if ($s->{type} =~ m/struct|union/) {
+			my $obj = findAPIObject($api, 'struct', $s->{name});
+			if ($obj->{default} eq 'all') {
+				push @list, @{$s->{fields}};
+			} else {
+				push @list, grep { findAPIField($obj, $_->{name}) } @{$s->{fields}};
+			}
+		} elsif ($s->{type} =~ m/func|call/) {
+			@list = @{$s->{arguments}};
+			push @list, $s->{result};
+		}
+
+		foreach $m (@list) {
+			my $type = $m->{type};
+
+			print " item $m->{name} '$type'\n" if $verbose;
+			if ($m->{ctype} =~ m/enum (.*)/) {
+				$type = "enum:$1";
+			}
+
+			push @roots,$type if $data{$type};
+		}
+	}
+
+	foreach $name (sort grep { m/:/ } keys %seen) {
+		print " $name\n";
+	}
+	print "\n";
+	return \%seen;
+}
+
+# find which api->thing->items applies to a given field name, if any
+sub findAPIItem {
+	my $api = shift;
+	my $type = shift;
+	my $target = shift;
+	my $mode = shift;
+	my $name = shift;
+
+	#print "search for $target.$name in $type.$mode\n";
+	# what about exclude?
+	foreach $obj ( @{$api->{$type}} ) {
+		if ($obj->{name} eq $target) {
+			#print " found $target\n";
+			foreach $inc (grep { $_->{mode} eq $mode } @{$obj->{items}}) {
+				#print "  check $inc->{match}\n";
+				return $inc if $name =~ m/$inc->{regex}/;
+			}
+		}
+	}
+}
+
+sub analyseAPI {
+	my $api = shift;
+
+	foreach $obj ( @{$api->{struct}}, @{$api->{library}}, @{$api->{func}}) {
+		$obj->{access} = 'rw';
+		$obj->{default} = 'all';
+		$obj->{rename} = $renameTable{'identity'};
+		$obj->{'func:rename'} = $renameTable{'identity'};
+		$obj->{'field:rename'} = $renameTable{'identity'};
+		foreach $o (@{$obj->{options}}) {
+			if ($o =~ m/^default=(none|all)$/) {
+				$obj->{default} = $1;
+			} elsif ($o =~ m/^access=([rwi]+)$/) {
+				$obj->{access} = $1;
+			} elsif ($o =~ m@^(rename|field:rename|func:rename)=(.*)@) {
+				my $target = $1;
+
+				if ($obj->{name} eq 'SwsContext') {
+					print "SwsContext rename = $o\n";
+				}
+
+				foreach $n (split /,/,$2) {
+					my $old = $obj->{$target};
+					my $new = $renameTable{$n};
+
+					if ($n =~ m@^s/(.*)/(.*)/$@) {
+						my $rx = qr/$1/;
+						my $rp = $2;
+						$obj->{$target} = sub { my $s=shift; $s = $old->($s); $s =~ s/$rx/$rp/; return $s;};
+					} elsif ($new) {
+						$obj->{$target} = sub { my $s=shift; $s = $old->($s); return $new->($s); };
+					}
+				}
+			}
+		}
+
+		my $defmode = $obj->{type} eq 'library' ? 'func' : 'field';
+
+		foreach $inc (@{$obj->{items}}) {
+			if ($inc->{match} =~ m@^(field|func|define|struct|enum):/(.*)/$@) {
+				$inc->{regex} = qr/$2/;
+				$inc->{mode} = $1; # ?? "$1-include";
+			} elsif ($inc->{match} =~ m@^(field|func|define|struct|enum):(.*)$@) {
+				$inc->{regex} = qr/^$2$/;
+				$inc->{mode} = $1;
+			} elsif ($inc->{match} =~ m@^/(.*)/$@) {
+				$inc->{regex} = qr/$1/;
+				$inc->{mode} = $defmode;
+			} else {
+				$inc->{regex} = qr/^$inc->{match}$/;
+				$inc->{mode} = $defmode;
+			}
+
+			$inc->{rename} = $renameTable{'identity'};
+			$inc->{scope} = 'static' if $obj->{type} eq 'library';
+
+			# maybe depends on mode above
+			foreach $o (@{$inc->{options}}) {
+				if ($o =~ m/^access=([rwi])+/) {
+					$inc->{access} = $1;
+				} elsif ($o =~ m/^rename=(.*)/) {
+					foreach $n (split /,/,$1) {
+						my $old = $inc->{rename};
+						my $new = $renameTable{$n};
+
+						if ($n =~ m@^s/(.*)/(.*)/$@) {
+							my $rx = qr/$1/;
+							my $rp = $2;
+							$inc->{rename} = sub { my $s=shift; $s = $old->($s); $s =~ s/$rx/$rp/; return $s;};
+						} elsif ($new) {
+							$inc->{rename} = sub { my $s=shift; $s = $old->($s); return $new->($s); };
+						} else {
+							my $x = $n;
+							$inc->{rename} = sub { return $x; };
+						}
+					}
+				} elsif ($o =~ m/^array-size=(.*)/) {
+					$inc->{'array_size'} = $1;
+				} elsif ($o =~ m/^array$/) {
+					$inc->{'array'} = 1;
+				} elsif ($o =~ m/^instance=(.*)/) {
+					$inc->{instance} = $1;
+				} elsif ($o =~ m/^static$/) {
+					$inc->{scope} = 'static';
+				} elsif ($o =~ m/^constructor=(.*)$/) {
+					$inc->{constructor} = $1;
+				} elsif ($o =~ m/^constructor-result=(.*)$/) {
+					$inc->{constructor_result} = $1;
+				} elsif ($o =~ m/^success=(.*)$/) {
+					$inc->{success} = $1;
+				}
+				# exclude mode, etc
+			}
+
+			$inc->{rename} = $obj->{"$inc->{mode}:rename"} if $inc->{rename} == $renameTable{'identity'} && $obj->{"$inc->{mode}:rename"};
+		}
+
+		if ($obj->{name} eq '<default>') {
+			$api->{"$obj->{type}:<default>"} = $obj;
+		}
+	}
+
+	$api->{'call:<default>'} = { rename => $renameTable{'identity'}, scope => 'static'} if !$api->{'call:<default>'};
 }
 
 # anonymous structs
@@ -104,7 +441,6 @@ sub analyseAndFixTypes {
 		my $s = $data{$n};
 		my @list;
 
-
 		if ($s->{type} =~ m/struct|union/) {
 			@list = @{$s->{fields}};
 		} elsif ($s->{type} =~ m/func|call/) {
@@ -127,7 +463,7 @@ sub analyseAndFixTypes {
 			}
 
 			# must be last
-			$a->{typeInfo} = queryTypeInfo($a);
+			$a->{typeInfo} = analyseTypeInfo($s, $a);
 		}
 	}
 
@@ -158,14 +494,18 @@ sub formatTypeLayout {
 	if ($m->{deref} =~ m/^(u64|u32):/) {
 		$desc .= "Memory.POINTER$withName";
 	} elsif ($m->{type} =~ m/^([iuf]\d+)$/) {
-		if ($m->{deref} =~ m/\[(\d*).*\]/) {
+		if ($m->{deref} =~ m/\[(\d*)u64:.*\]/) {
+			$desc .= "MemoryLayout.sequenceLayout($1, Memory.POINTER)$withName";
+		} elsif ($m->{deref} =~ m/\[(\d*).*\]/) {
 			$desc .= "MemoryLayout.sequenceLayout($1, Memory.".uc($typeSizes{$m->{type}}).")$withName";
 		} else {
 			$desc .= 'Memory.'.uc($typeSizes{$m->{type}})."$withName";
 		}
 	} elsif ($m->{type} =~ m/^(struct|union):(.*)/) {
 		my $type = $2;
-		if ($m->{deref} =~ m/\[(\d*).*\]/) {
+		if ($m->{deref} =~ m/\[(\d*)u64:.*\]/) {
+			$desc .= "MemoryLayout.sequenceLayout($1, Memory.POINTER)$withName";
+		} elsif ($m->{deref} =~ m/\[(\d*).*\]/) {
 			$desc .= "MemoryLayout.sequenceLayout($1, $type.LAYOUT)$withName";
 		} else {
 			$desc .= "$type.LAYOUT$withName";
@@ -222,9 +562,19 @@ sub formatSignature {
 	return $desc;
 }
 
-sub queryTypeInfo {
-	my $m = shift @_;
+# TODO: perhaps ByteArray should just be MemorySegment, kinda painful to wrap them all the time
+sub analyseTypeInfo {
+	my $s = shift;
+	my $m = shift;
 	my $info = {};
+	my $inc;
+
+	#print " query $s->{name} $s->{type} '$m->{name}', '$m->{type}'\n";
+	if ($s->{type} eq 'struct') {
+		$inc = findAPIItem($api, 'struct', $s->{name}, 'field', $m->{name});
+	} elsif ($s->{type} eq 'func') {
+		$inc = findAPIItem($api, 'func', $s->{name}, 'field', $m->{name});
+	}
 
 	# default for everything not specifically handled
 	$info->{carrier} = "MemoryAddress";
@@ -234,21 +584,43 @@ sub queryTypeInfo {
 		# This is a function pointer, type must be type = 'call:.*'
 		if ($m->{type} =~ m/^call:(.*)/) {
 			$info->{type} = "Memory.FunctionPointer<$1>";
-			$info->{create} = "$1.downcall(\${result}, scope())";
+			$info->{create} = "$1.downcall(\${result}, \${scope})";
 		} else {
 			die();
 		}
 	} elsif ($m->{type} =~ m/^([iuf]\d+)$/) {
-		if ($m->{deref} =~ m/\[(\d*).*\]/) {
+		if ($m->{deref} =~ m/\[(\d*)u64:.*\]/) {
+			$info->{byValue} = 1;
+			$info->{type} = "Memory.PointerArray";
+			$info->{create} = $info->{type}.".create(\${result})";
+		} elsif ($m->{deref} =~ m/\[(\d*).*\]/) {
+			# TODO: some mode thing rather than byvalue?
 			$info->{byValue} = 1;
 			$info->{type} = "Memory.".ucfirst($typeSizes{$m->{type}})."Array";
 			$info->{create} = $info->{type}.".create(\${result})";
 		} elsif ($m->{deref} =~ m/^(u64:u64:|u32:u32:)/) {
 			$info->{type} = "Memory.PointerArray";
-			$info->{create} = $info->{type}.".create(\${result})";
+			$info->{create} = $info->{type}.".createArray(\${result}, Long.MAX_VALUE, \${scope})";
 		} elsif ($m->{deref} =~ m/^(u64:|u32:)/) {
-			$info->{type} = "Memory.".ucfirst($typeSizes{$m->{type}})."Array";
-			$info->{create} = $info->{type}.".create(\${result})";
+			# assume any char * is a string unless an array-size or array is specified
+			if ($inc->{array}) {
+				$info->{type} = "Memory.".ucfirst($typeSizes{$m->{type}})."Array";
+				$info->{create} = $info->{type}.".createArray(\${result}, Long.MAX_VALUE, \${scope})";
+			} elsif ($inc->{array_size}) {
+				$info->{type} = "Memory.".ucfirst($typeSizes{$m->{type}})."Array";
+				$info->{create} = $info->{type}.".createArray(\${result}, \${array_size}, \${scope})";
+			} elsif ($typeSizes{$m->{type}} eq 'byte') {
+				$info->{type} = 'String';
+				$info->{resolve} = "(Addressable)Memory.address(frame.copy(\${value}))";
+				$info->{create} = "(\${result}).getUtf8String(0)";
+				$info->{resolveFrame} = 1;
+				# for a function or a constructor that uses this element
+				$s->{resolveFrame} = 1;
+			} else {
+				# ideally length 0 but panama-foreign doesn't grok that so fuckit
+				$info->{type} = "Memory.".ucfirst($typeSizes{$m->{type}})."Array";
+				$info->{create} = $info->{type}.".createArray(\${result}, Long.MAX_VALUE, \${scope})";
+			}
 		} else {
 			$info->{type} = $typeSizes{$m->{type}};
 			$info->{carrier} = $typeSizes{$m->{type}};
@@ -257,23 +629,42 @@ sub queryTypeInfo {
 		}
 	} elsif ($m->{type} =~ m/^(struct|union):(.*)/) {
 		my $type = $2;
-		if ($m->{deref} =~ m/\[(\d*).*\]/) {
-			$info->{type} = $type;
-			$info->{create} = $info->{type}.".createArray($1, \${result}, scope())";
+		if ($m->{deref} =~ m/\[(\d*)(.*)\]/) {
+			$info->{byValue} = 1;
+			# handle  'type name[x]' and  'type *name[x]'
+			my $count = $1;
+			my $deref = $2;
+			if ($deref =~ m/^u64:u64:/) {
+				die("can't handle double-deref array");
+			} elsif ($deref =~ m/^u64:/) {
+				$info->{type} = "Memory.HandleArray<$type>";
+				$info->{create} = "Memory.HandleArray.create(\${result}, $type\:\:create, \${scope})";
+			} else {
+				$info->{type} = $type;
+				#$info->{create} = $info->{type}.".create(\${result}, \${scope})";
+				$info->{create} = $info->{type}.".create(\${result})";
+			}
 		} elsif ($m->{deref} =~ m/^(u64:u64:|u32:u32:)/) {
-			$info->{type} = "Memory.PointerArray";
-			$info->{create} = $info->{type}.".create(\${result})";
+			# this assumes ** is as far as it gets
+			$info->{type} = "Memory.HandleArray<$type>";
+			if ($inc->{array_size}) {
+				$info->{create} = "Memory.HandleArray.createArray(\${result}, \${array_size}, $type\:\:create, \${scope})";
+			} else {
+				$info->{create} = "Memory.HandleArray.createArray(\${result}, Long.MAX_VALUE, $type\:\:create, \${scope})";
+			}
 		} elsif ($m->{deref} =~ m/^(u64:|u32:)/) {
 			$info->{type} = $type;
-			$info->{create} = $info->{type}.".create(\${result}, scope())";
+			$info->{create} = $info->{type}.".create(\${result}, \${scope})";
 		} else {
+			# FIXME: change this to a reftype or something
+			$info->{byValue} = 1;
 			$info->{type} = $type;
 			$info->{create} = $info->{type}.".create(\${result})";
 		}
 	} elsif ($m->{type} eq "void") {
 		if ($m->{deref} =~ m/^(u64:u64:|u32:u32:)/) {
 			$info->{type} = "Memory.PointerArray";
-			$info->{create} = $info->{type}.".create(\${result})";
+			$info->{create} = $info->{type}.".createArray(\${result}, Long.MAX_VALUE, \${scope})";
 		} elsif ($m->{deref} =~ m/^(u64:|u32:)/) {
 			$info->{type} = "MemoryAddress";
 			$info->{create} = "\${result}";
@@ -291,45 +682,160 @@ sub queryTypeInfo {
 	return $info;
 }
 
-sub formatFunction {
-	my $c = shift @_;
+# TODO: see if this can be merged with formatFunction
+# TODO: should constructor take a resourcescope always?
+# TODO: exceptions for errors
+sub formatConstructor {
+	my $c = shift;
+	my $inc = shift;
 	my @arguments = @{$c->{arguments}};
 	my $result = $c->{result};
 	my $desc;
 	my $index = 0;
-	my $desc;
+	my $count = 0;
+	my $desc = "";
 	my $name = $c->{name};
-
 	my $rtype = $result->{typeInfo}->{type};
+	my $otype = $data{$arguments[$inc->{constructor_result}]->{type}}->{name};
 
-	#print Dumper($c);
 
-	$desc = $rtype;
+	$name = $inc->{rename}->($name);
+
+	$desc .= "static " if $inc->{scope} eq 'static';
+	$desc .= $otype;
 	$desc .= " $name(";
 
 	for $m (@arguments) {
-		$desc .= ", " if ($index++ > 0);
-		$desc .= $m->{typeInfo}->{type};
-		$desc .= " $m->{name}"
+		if (($inc->{constructor} && $index != $inc->{constructor_result})) {
+			$desc .= ", " if ($count++ > 0);
+			$desc .= $m->{typeInfo}->{type};
+			$desc .= " $m->{name}"
+		}
+		$index++;
+	}
+	if ($inc->{constructor}) {
+		$desc .= ", " if ($count++ > 0);
+		$desc .= "ResourceScope scope";
 	}
 	$desc .=") {\n ";
+
+	$desc .= "$result->{typeInfo}->{carrier} res\$value;\n" if ($rtype ne "void");
+
+	$desc .= " try ";
+	$desc .= "(Frame frame = Memory.createFrame()) " if ($c->{resolveFrame});
+	$desc .= "{\n";
+	$desc .= "   Memory.HandleArray<$otype> res\$holder = Memory.HandleArray.createArray(1, frame, $otype\:\:create, scope);\n" if ($inc->{constructor});
+	$desc .= "   res\$value = ($result->{typeInfo}->{carrier})" if ($rtype ne "void");
+	$desc .= "   " if ($rtype eq "void");
+
 	$index = 0;
+	$desc .= "$c->{name}\$FH.invokeExact(\n   ";
+	for $m (@arguments) {
+		my $resolve = $m->{typeInfo}->{resolve};
+
+		if ($inc->{constructor} && $index == $inc->{instance}) {
+			$desc .= ",\n    " if ($index++ > 0);
+			$desc .= "(Addressable)res\$holder.address()";
+		} elsif ($inc->{scope} ne 'static' && $index == $inc->{instance}) {
+			$desc .= ",\n    " if ($index++ > 0);
+			$desc .= "(Addressable)address()";
+		} else {
+			$desc .= ",\n    " if ($index++ > 0);
+
+			if ($resolve) {
+				$resolve =~ s/\$\{value\}/$m->{name}/g;
+				$desc .= $resolve;
+			} else {
+				$desc .= "$m->{name}";
+			}
+		}
+	}
+	$desc .= ");\n";
+
+	if ($rtype ne "void" && defined $inc->{success}) {
+	# 	my $create = $result->{typeInfo}->{create};
+
+	# 	# ooh, templates could insert other arguments or values as well?
+	# 	$create =~ s/\$\{result\}/res\$value/;
+	# 	if ($inc->{scope} eq 'static') {
+	# 		$create =~ s/\$\{scope\}/ResourceScope.globalScope()/;
+	# 	} else {
+	# 		$create =~ s/\$\{scope\}/scope()/;
+	# 	}
+
+		foreach $code (split /,/,$inc->{success}) {
+			$desc .= "   if (res\$value == $code) return res\$holder.getAtIndex(0);\n";
+		}
+	} else {
+		$desc .= "   return res\$holder.getAtIndex(0);\n";
+	}
+	# throw Error()?
+	$desc .= " } catch (Throwable t) { throw new RuntimeException(t); }\n";
+
+	# throw failures here based on res$value
+	$desc .= " return null;\n";
+
+	$desc .="}";
+
+	#print "$desc\n";
+	return $desc;
+}
+
+sub formatFunction {
+	my $c = shift;
+	my $inc = shift;
+	my @arguments = @{$c->{arguments}};
+	my $result = $c->{result};
+	my $desc;
+	my $index = 0;
+	my $count = 0;
+	my $desc = "";
+	my $name = $c->{name};
+	my $rtype = $result->{typeInfo}->{type};
+
+	if ($inc->{constructor}) {
+		return formatConstructor($c, $inc);
+	}
+
+	$name = $inc->{rename}->($name);
+
+	$desc .= "static " if $inc->{scope} eq 'static';
+	$desc .= $rtype;
+	$desc .= " $name(";
+
+	for $m (@arguments) {
+		if ($inc->{scope} eq 'static' || $index != $inc->{instance}) {
+			$desc .= ", " if ($count++ > 0);
+			$desc .= $m->{typeInfo}->{type};
+			$desc .= " $m->{name}"
+		}
+		$index++;
+	}
+	$desc .=") {\n ";
 
-	$desc .= "try {\n";
+	$desc .= "try ";
+	$desc .= "(Frame frame = Memory.createFrame()) " if ($c->{resolveFrame});
+	$desc .= "{\n";
 	$desc .= "  $result->{typeInfo}->{carrier} res\$value = ($result->{typeInfo}->{carrier})" if ($rtype ne "void");
 	$desc .= "  " if ($rtype eq "void");
 
-	$desc .= "$name\$FH.invokeExact(\n   ";
+	$index = 0;
+	$desc .= "$c->{name}\$FH.invokeExact(\n   ";
 	for $m (@arguments) {
 		my $resolve = $m->{typeInfo}->{resolve};
 
-		$desc .= ",\n   " if ($index++ > 0);
-
-		if ($resolve) {
-			$resolve =~ s/\$\{value\}/$m->{name}/g;
-			$desc .= $resolve;
+		if ($inc->{scope} ne 'static' && $index == $inc->{instance}) {
+			$desc .= ",\n   " if ($index++ > 0);
+			$desc .= "(Addressable)address()";
 		} else {
-			$desc .= "$m->{name}"
+			$desc .= ",\n   " if ($index++ > 0);
+
+			if ($resolve) {
+				$resolve =~ s/\$\{value\}/$m->{name}/g;
+				$desc .= $resolve;
+			} else {
+				$desc .= "$m->{name}";
+			}
 		}
 	}
 	$desc .= ");\n";
@@ -338,7 +844,12 @@ sub formatFunction {
 		my $create = $result->{typeInfo}->{create};
 
 		# ooh, templates could insert other arguments or values as well?
-		$create =~ s/\${result\}/res\$value/;
+		$create =~ s/\$\{result\}/res\$value/;
+		if ($inc->{scope} eq 'static') {
+			$create =~ s/\$\{scope\}/ResourceScope.globalScope()/;
+		} else {
+			$create =~ s/\$\{scope\}/scope()/;
+		}
 
 		$desc .= "  return $create;\n";
 	}
@@ -351,9 +862,10 @@ sub formatFunction {
 }
 
 # create an interface for function pointers
-# FiXME: this should be exportCallback to a file
+# FiXME: this should be exportCallback to a file?
 sub formatCallback {
-	my $c = shift @_;
+	my $c = shift;
+	my $obj = shift;
 	my @arguments = @{$c->{arguments}};
 	my $result = $c->{result};
 	my $desc;
@@ -421,6 +933,7 @@ sub formatCallback {
 		my $create = $m->{typeInfo}->{create};
 
 		$create =~ s/\$\{result\}/$m->{name}/g;
+		$create =~ s/\$\{scope\}/scope/g;
 
 		$desc .= ",\n    " if ($index++ > 0);
 		$desc .= "$create";
@@ -450,12 +963,13 @@ sub formatCallback {
 	$desc .= "   symbol,\n";
 
 	# HACK: this is basically the same as any function call, just patch in the changes for now
-	$tmp = formatFunction($c);
+	$tmp = formatFunction($c, $obj);
 
 	$tmp =~ s/^(.*) ($name)\(/(/;
 	$tmp =~ s/\) \{/) -> {/;
 	$tmp =~ s/^/   /mg;
 	$desc .= $tmp;
+
 	$desc .= "\n";
 	$desc .= "  );\n";
 	$desc .= " }\n";
@@ -591,72 +1105,91 @@ sub formatLayout {
 }
 
 sub formatGetSet {
-	my $s = shift @_;
-	my $m = shift @_;
+	my $s = shift;
+	my $m = shift;
+	my $rename = shift;
+	my $access = shift;
+	my $inc = shift;
 	my $desc = "";
 	my $info = $m->{typeInfo};
-	my $Name = ucfirst($m->{name});
+	my $Name = ucfirst($rename);
 	my $tmp;
 
 	# info -> needsalloc?
 
-	# TODO: String
 	# TODO: embedded arrays are quite different setup
 
 	if ($info->{byValue}) {
 		$tmp = $info->{create};
 		$tmp =~ s/\$\{result\}/segment/g;
+		$tmp =~ s/\$\{scope\}/scope()/g;
 
 		$desc .= " public $info->{type} get$Name() {\n";
-		$desc .= "  MemoryLayout.PathElement pe = MemoryLayout.PathElement.groupElement(\"$m->{name}\");\n";
-		$desc .= "  MemorySegment segment = this.segment.asSlice(LAYOUT.byteOffset(pe), LAYOUT.select(pe).byteSize());\n";
+		#$desc .= "  MemoryLayout.PathElement pe = MemoryLayout.PathElement.groupElement(\"$m->{name}\");\n";
+		#$desc .= "  MemorySegment segment = this.segment.asSlice(LAYOUT.byteOffset(pe), LAYOUT.select(pe).byteSize());\n";
+		$desc .= "  MemorySegment segment = this.segment.asSlice($m->{name}\$byteOffset, $m->{name}\$byteSize);\n";
 		$desc .= "  return $tmp;\n";
 		$desc .= " }\n";
 
-		$desc .= " public $info->{type} get$Name"."At(long index) {\n";
-		$desc .= "  MemorySegment segment = this.segment.asSlice(LAYOUT.byteSize() * index, LAYOUT.byteSize());\n";
-		$desc .= "  MemoryLayout.PathElement pe = MemoryLayout.PathElement.groupElement(\"$m->{name}\");\n";
-		$desc .= "  segment = this.segment.asSlice(LAYOUT.byteOffset(pe), LAYOUT.select(pe).byteSize());\n";
-		$desc .= "  return $tmp;\n";
-		$desc .= " }\n";
-	} else {
+		if ($access =~ m/i/) {
+			$desc .= " public $info->{type} get$Name"."At(long index) {\n";
+			#$desc .= "  MemorySegment segment = this.segment.asSlice(LAYOUT.byteSize() * index, LAYOUT.byteSize());\n";
+			#$desc .= "  MemoryLayout.PathElement pe = MemoryLayout.PathElement.groupElement(\"$m->{name}\");\n";
+			#$desc .= "  segment = this.segment.asSlice(LAYOUT.byteOffset(pe), LAYOUT.select(pe).byteSize());\n";
+			$desc .= "  MemorySegment segment = this.segment.asSlice(LAYOUT.byteSize() * index + $m->{name}\$byteOffset, $m->{name}\$byteSize);\n";
+			$desc .= "  return $tmp;\n";
+			$desc .= " }\n";
+		}
+	} elsif ($access =~ /r/) {
 		$tmp = $info->{create};
+
 		$tmp =~ s/\$\{result\}/($info->{carrier})$m->{name}\$VH.get(segment)/g;
+		$tmp =~ s/\$\{scope\}/scope()/g;
+		# fixme: lookup type of array size? somewhere?  doesn't matter i think
+		$tmp =~ s/\${array_size}/(long)$inc->{array_size}\$VH.get(segment)/g;
 
 		$desc .= " public $info->{type} get$Name() {\n";
 		$desc .= "  return $tmp;\n";
 		$desc .= " }\n";
 
-		$desc .= " public $info->{type} get$Name"."At(long index) {\n";
-		$desc .= "  MemorySegment segment = this.segment.asSlice(LAYOUT.byteSize() * index, LAYOUT.byteSize());\n";
-		$desc .= "  return $tmp;\n";
-		$desc .= " }\n";
+		if ($access =~ m/i/) {
+			$desc .= " public $info->{type} get$Name"."At(long index) {\n";
+			$desc .= "  MemorySegment segment = this.segment.asSlice(LAYOUT.byteSize() * index, LAYOUT.byteSize());\n";
+			$desc .= "  return $tmp;\n";
+			$desc .= " }\n";
+		}
 	}
 
-	if (!($m->{deref} =~ m/\[(\d*).*\]/)) {
+	if ($access =~ m/w/ && !$info->{byValue}) {
 		$tmp = $info->{resolve};
 		$tmp =~ s/\$\{value\}/value/g;
 
 		$desc .=  " public void set$Name($info->{type} value) {\n";
+		$desc .=  " try (Frame frame = Memory.createFrame()) {\n" if ($info->{resolveFrame});
 		$desc .=  "  $m->{name}\$VH.set(segment, $tmp);\n";
+		$desc .=  " }\n" if ($info->{resolveFrame});
 		$desc .=  " }\n";
 
-		$desc .=  " public void set$Name"."At(long index, $info->{type} value) {\n";
-		$desc .=  "  MemorySegment segment = this.segment.asSlice(LAYOUT.byteSize() * index, LAYOUT.byteSize());\n";
-		$desc .=  "  $m->{name}\$VH.set(segment, $tmp);\n";
-		$desc .=  " }\n";
+		if ($access =~ m/i/) {
+			$desc .=  " public void set$Name"."At(long index, $info->{type} value) {\n";
+			$desc .=  " try (Frame frame = Memory.createFrame()) {\n" if ($info->{resolveFrame});
+			$desc .=  "  MemorySegment segment = this.segment.asSlice(LAYOUT.byteSize() * index, LAYOUT.byteSize());\n";
+			$desc .=  "  $m->{name}\$VH.set(segment, $tmp);\n";
+			$desc .=  " }\n" if ($info->{resolveFrame});
+			$desc .=  " }\n";
+		}
 	}
 
-	# indexed
-
 	return $desc;
 }
 
 sub exportStruct {
-	my $f = shift @_;
-	my $s = shift @_;
+	my $f = shift;
+	my $s = shift;
+	my $obj = shift;
 	my @fields = @{$s->{fields}};
 	my $isHandle = $s->{size} == 0;
+	my $doArray = $obj->{access} =~ m/i/;
 	#my @functions = @{shift @_};
 
 	print $f "package $package;\n" if $package;
@@ -673,11 +1206,13 @@ sub exportStruct {
 		print $f " private $s->{name}(MemorySegment segment) { this.segment = segment; }\n";
 		print $f " public static $s->{name} create(MemorySegment segment) { return new $s->{name}(segment); }\n";
 		print $f " public static $s->{name} create(MemoryAddress address, ResourceScope scope) {\n";
-		print $f "  return create(MemorySegment.ofAddress(address, LAYOUT.byteSize(), scope));\n";
-		print $f " }\n";
-		print $f " public static $s->{name} createArray(MemoryAddress address, long size, ResourceScope scope) {\n";
-		print $f "  return create(MemorySegment.ofAddress(address, size * LAYOUT.byteSize(), scope));\n";
+		print $f "  return MemoryAddress.NULL != address ? create(MemorySegment.ofAddress(address, LAYOUT.byteSize(), scope)) : null;\n";
 		print $f " }\n";
+		if ($doArray) {
+			print $f " public static $s->{name} createArray(MemoryAddress address, long size, ResourceScope scope) {\n";
+			print $f "  return MemoryAddress.NULL != address ? create(MemorySegment.ofAddress(address, size * LAYOUT.byteSize(), scope)) : null;\n";
+			print $f " }\n";
+		}
 		print $f " public static $s->{name} create(Frame frame) { return create(frame.allocate(LAYOUT)); }\n";
 		print $f " public static $s->{name} create(ResourceScope scope) { return create(MemorySegment.allocateNative(LAYOUT, scope)); }\n";
 		print $f " public MemoryAddress address() { return segment.address(); }\n";
@@ -687,91 +1222,81 @@ sub exportStruct {
 		print $f " MemoryAddress address;\n";
 		print $f " ResourceScope scope;\n";
 		# constructors
-		print $f " private $s->{name}(MemoryAddress address, ResourceScope scope) { this.address = address; this.scope = scope}\n";
-		print $f " public static $s->{name} create(MemoryAddress address) { return new $s->{name}(address); }\n";
+		print $f " private $s->{name}(MemoryAddress address, ResourceScope scope) { this.address = address; this.scope = scope;}\n";
+		print $f " public static $s->{name} create(MemoryAddress address, ResourceScope scope) { return MemoryAddress.NULL != address ? new $s->{name}(address, scope) : null; }\n";
 		print $f " public MemoryAddress address() { return address; }\n";
 		print $f " public ResourceScope scope() { return scope; }\n";
 	}
 
-	# FIXME: use typeInfo
-	# TODO: indexed accessors
-	# accessors
-	if (1) {
-		foreach $m (@fields) {
-			print $f formatGetSet($s, $m);
-		}
-	} else {
-		foreach $m (@fields) {
-			my $Name = ucfirst($m->{name});
+	my %seen;
 
-			print $f " // [$m->{deref}] [$m->{type}] [$m->{ctype}]\n";
+	# Any defines
+	foreach $inc (grep { $_->{mode} eq 'define' } @{$obj->{items}}) {
+		my $def = $data{$inc->{match}};
 
-			if ($m->{deref} =~ m/^(u64:|u32:)\(/) {
-				# This is a function pointer, type must be type = 'call:.*'
+		die ("unknown define $inc->{match} in $s->{name}\n") if !$def;
 
-				if ($m->{type} =~ m/^call:(.*)/) {
-					my $jtype = $1;
+		delete $toDump->{$inc->{match}};
 
-					$jtype =~ s/(.*)\((.*)\)(.*)/Call$1_$2_$3/;
+		foreach $m (@{$def->{values}}) {
+			print $f " /**\n ($m->{comment}) */\n" if ($m->{comment});
+			print $f " public static final $defineType{$m->{type}} $m->{name} = $definePrefix{$m->{type}}$m->{value}$defineSuffix{$m->{type}};\n";
+		}
+	}
 
-					print $f " public Memory.FunctionPointer<$jtype> get$Name() {\n";
-					print $f "  // FIXME: better scope\n";
-					print $f "  return $jtype.downcall((MemoryAddress)$m->{name}\$VH.get(segment), scope());\n";
-					print $f " }\n";
+	# TODO: any enums we want to include here I suppose?
 
-					print $f " public void set$Name(Memory.FunctionPointer<$jtype> value) {\n";
-					print $f "  $m->{name}\$VH.set(segment, Memory.address(value));\n";
-					print $f " }\n";
-				}
-			} elsif ($m->{deref} =~ m/^(u64:|u32:)/) {
-				# all other pointer types require extra context, e.g. a length or type
-				print $f " public MemoryAddress get$Name() {\n";
-				print $f "  return (MemoryAddress)$m->{name}\$VH.get(segment);\n";
-				print $f " }\n";
-
-				# FIXME: set could use the type though
-				print $f " public void set$Name(MemoryAddress value) {\n";
-				print $f "  $m->{name}\$VH.set(segment, Memory.address(value));\n";
-				print $f " }\n";
-			} elsif ($m->{type} eq "bitfield") {
-				# TODO
-			} elsif ($m->{type} =~ m/^struct|union:(.*)$/) {
-				my $jtype = $1;
-
-				$jtype = "Memory.HandleArray<$jtype>" if ($data{$m->{type}}->{size} == 0);
-
-				# embedded type including arrays
-				print $f " public $jtype get$Name() {\n";
-				print $f "  MemoryLayout.PathElement pe = MemoryLayout.PathElement.groupElement(\"$m->{name}\");\n";
-				print $f "  MemorySegment seg = segment.asSlice(LAYOUT.byteOffset(pe), LAYOUT.select(pe).byteSize());\n";
-				print $f "  return $jtype.create(seg, $jtype::new);\n" if ($data{$m->{type}}->{size} == 0);
-				print $f "  return $jtype.create(seg);\n" if ($data{$m->{type}}->{size} != 0);
-				print $f " }\n";
-			} elsif ($m->{type} =~ m/^[uif]\d+$/) {
-				my $jtype = $typeSizes{$m->{type}};
-				my $Jtype = ucfirst($jtype);
-				my $JTYPE = uc($jtype);
-
-				if ($m->{deref} =~ m/\[(\d*).*\]/) {
-					# array type
-					print $f " public Memory.$Jtype"."Array get$Name() {\n";
-					print $f "  MemoryLayout.PathElement pe = MemoryLayout.PathElement.groupElement(\"$m->{name}\");\n";
-					print $f "  MemorySegment seg = segment.asSlice(LAYOUT.byteOffset(pe), LAYOUT.select(pe).byteSize());\n";
-					print $f "  return new Memory.$Jtype"."Array(seg);\n";
-					print $f " }\n";
-				} else {
-					# primitive type
-					print $f " public $jtype get$Name() {\n";
-					print $f "  return ($jtype)$m->{name}\$VH.get(segment);\n";
-					print $f " }\n";
-
-					print $f " public void set$Name($jtype value) {\n";
-					print $f "  $m->{name}\$VH.set(segment, ($jtype)value);\n";
-					print $f " }\n";
-				}
+	# Accessors
+	foreach $m (@fields) {
+		my $access = $obj->{access};
+		my $rename = $obj->{'field:rename'};
+		my $matches = 0;
+		my $matchinc;
+
+		# check for match
+		foreach $inc (grep { $_->{mode} eq 'field' } @{$obj->{items}}) {
+			$matches = $m->{name} =~ m/$inc->{regex}/;
+
+			if ($matches) {
+				$access = $inc->{access} if $inc->{access};
+				$rename = $inc->{rename} if $inc->{rename} != $renameTable{'identity'};
+				$matchinc = $inc;
+				last;
 			}
-			# struct				print $f " MemoryLayout.PathElement pe = MemoryLayout.PathElement.groupElement(\"$m->{name}\");\n";
+		}
+
+		my $output = $matches || ($obj->{default} eq 'all');
+
+		if ($output) {
+			my $name = $rename ? $rename->($m->{name}) : $m->{name};
 
+			print $f formatGetSet($s, $m, $name, $access, $matchinc);
+		}
+	}
+
+	# Functions
+	foreach $inc (grep { $_->{mode} eq 'func' } @{$obj->{items}}) {
+		my @list;
+
+		print "$obj->{name} match $inc->{match} regex $inc->{regex}\n" if $verbose;
+
+		if ($data{$inc->{match}}) {
+			push @list, $data{$inc->{match}};
+		} else {
+			@list = grep { $_->{name} =~ m/$inc->{regex}/ } values %data;
+		}
+
+		foreach $c (@list) {
+			my $tmp;
+
+			next if $seen{$c->{name}}++;
+
+			print $f " static final MethodHandle $c->{name}\$FH = Memory.downcall(\"$c->{name}\",\n";
+			$tmp = formatFunctionDescriptor($c);
+			print $f "$tmp);\n";
+
+			$tmp = formatFunction($c, $inc);
+			print $f 'public '.$tmp."\n\n";
 		}
 	}
 
@@ -780,11 +1305,59 @@ sub exportStruct {
 		print $f "static final GroupLayout LAYOUT = ".formatLayout($s).";\n";
 
 		foreach $m (@fields) {
-			next if ($m->{typeInfo}->{byValue});
-			print $f " static final VarHandle $m->{name}\$VH = LAYOUT.varHandle(MemoryLayout.PathElement.groupElement(\"$m->{name}\"));\n";
+			print $f " // type='$m->{type}' deref='$m->{deref}'  info->type ='$m->{typeInfo}->{type}'\n";
+			if ($m->{typeInfo}->{byValue}) {
+				print $f " static final long $m->{name}\$byteOffset = "
+					." LAYOUT.byteOffset(MemoryLayout.PathElement.groupElement(\"$m->{name}\"));\n";
+				print $f " static final long $m->{name}\$byteSize = "
+					."LAYOUT.select(MemoryLayout.PathElement.groupElement(\"$m->{name}\")).byteSize();\n";
+			} else {
+				print $f " static final VarHandle $m->{name}\$VH = "
+					."LAYOUT.varHandle(MemoryLayout.PathElement.groupElement(\"$m->{name}\"));\n";
+			}
 		}
 	}
 
+	# verification?
+	if (!$isHandle) {
+		print $f <<END;
+		static void check\$offset(String name, long bitoffset) {
+			long byteoffset = bitoffset/8;
+			long offset = LAYOUT.byteOffset(MemoryLayout.PathElement.groupElement(name));
+			if (offset != byteoffset)
+				throw new AssertionError(String.format("%s.offset %d != %d", name, byteoffset, offset), null);
+		}
+		{
+END
+		foreach $m (@fields) {
+			print $f "check\$offset(\"$m->{name}\", $m->{offset});\n";
+		}
+my $bytes = $s->{size}/8;
+	print $f <<END;
+		if (LAYOUT.byteSize() != $bytes)
+			throw new AssertionError(String.format("$s->{name}.sizeof = %d != $bytes", LAYOUT.byteSize()), null);
+	}
+END
+	}
+	print $f "}\n";
+}
+
+# file,enum
+sub exportEnum {
+	my $f = shift;
+	my $s = shift;
+	my @values = @{$s->{values}};
+	my $jtype = $typeSizes{$s->{value_type}};
+	my $prefix = $definePrefix{$s->{value_type}};
+	my $suffix = $definePrefix{$s->{value_type}};
+	print $f "package $package;\n" if $package;
+
+	print $f "public interface $s->{name} {\n";
+
+	foreach $v (@values) {
+		print $f " public static final $jtype $v->{name} = $prefix$v->{value}$suffix;\n";
+	}
+
 	print $f "}\n";
 }
 
@@ -825,22 +1398,46 @@ sub nameToPath {
 	return $name;
 }
 
-foreach $x (grep { m/^(struct|union):/ } sort keys %data) {
-	my $s = $data{$x};
-	my $path = nameToPath($output, "$package.$s->{name}");
+#print "api\n";
+#print Dumper($api);
 
-	open (my $f, ">", $path) || die ("Cannot open '$path' for writing");
+# Dump struct type
+foreach $obj ( @{$api->{struct}} ) {
+	my @list;
 
-	exportStruct($f, $s);
+	next if $obj->{name} eq '<default>';
 
-	close $f;
-}
+	if ($obj->{name} =~ m@/(.*)/@) {
+		my $rx = qr/struct:$1/;
 
-foreach $x (grep { m/^call:/ } sort keys %data) {
-	my $c = $data{$x};
-	my $name = $c->{name};
+		@list = map { s/struct://; $_ } grep { $_ =~ m/$rx/ } keys %data;
+	} else {
+		push @list, $obj->{name};
+	}
 
-	my $path = nameToPath($output, "$package.$name");
+	foreach $name (@list) {
+		my $path = nameToPath($output, "$package.$name");
+		my $s = $data{"struct:$name"};
+
+		#print Dumper($obj);
+
+		delete $toDump->{"struct:$name"};
+
+		if ($s) {
+			open (my $f, ">", $path) || die ("Cannot open '$path' for writing");
+
+			exportStruct($f, $s, $obj);
+
+			close $f;
+		} else {
+			print "No struct $name\n";
+		}
+	}
+}
+
+# Dump library type
+foreach $lib ( @{$api->{library}} ) {
+	my $path = nameToPath($output, "$package.$lib->{name}");
 
 	open (my $f, ">", $path) || die ("Cannot open '$path' for writing");
 
@@ -848,22 +1445,57 @@ foreach $x (grep { m/^call:/ } sort keys %data) {
 	print $f "import jdk.incubator.foreign.*;\n";
 	print $f "import java.lang.invoke.*;\n";
 
-	print $f formatCallback($c);
+	print $f "public class $lib->{name} {\n";
+
+	print $f " static ResourceScope scope() { return ResourceScope.globalScope(); }\n";
+
+	# scan for matches
+	foreach $inc (@{$lib->{items}}) {
+		if ($inc->{mode} eq 'func') {
+			my @list = grep { $_->{type} eq $inc->{mode} && $_->{name} =~ m/$inc->{regex}/ } values %data;
+			foreach $c (@list) {
+				my $tmp;
+
+				print $f " static final MethodHandle $c->{name}\$FH = Memory.downcall(\"$c->{name}\",\n";
+				$tmp = formatFunctionDescriptor($c);
+				print $f $tmp.");\n";
+
+				$tmp = formatFunction($c, $inc);
+				print $f "public ";
+				print $f $tmp."\n\n";
+			}
+		} elsif ($inc->{mode} eq 'define') {
+			my @list = grep { $_->{type} eq $inc->{mode} && $_->{name} =~ m/$inc->{regex}/ } values %data;
+			foreach $c (@list) {
+				delete $toDump->{"define:$c->{name}"};
+				foreach $m (@{$c->{fields}}) {
+					print $f " /**\n ($m->{comment}) */\n" if ($m->{comment});
+					print $f " public static final $defineType{$m->{type}} $m->{name} = $definePrefix{$m->{type}}$m->{value}$defineSuffix{$m->{type}};\n";
+				}
+			}
+		}
+	}
+
+	print $f "}\n";
 
 	close $f;
 }
 
-# just quick and dirty for now
-# may want a non-static version with a specific scope?
-{
-	my @functions = grep { /^func:/ } keys %data;
-	my $lib = {
-		name => "APILib",
-		functions => \@functions
-	};
+print "remaining dependent types\n";
+foreach $k (sort grep { !m/func:/ } keys %{$toDump}) {
+	print " $k\n";
+}
+print "\n";
 
+# Calls referenced
+foreach $k (sort keys %{$toDump}) {
+	next if (!($k =~ m/call:(.+)/));
 
-	my $path = nameToPath($output, "$package.$lib->{name}");
+	my $name = $1;
+	my $c = $data{$k};
+	my $obj = findAPIObject($api, 'call', $name);
+
+	my $path = nameToPath($output, "$package.$name");
 
 	open (my $f, ">", $path) || die ("Cannot open '$path' for writing");
 
@@ -871,23 +1503,117 @@ foreach $x (grep { m/^call:/ } sort keys %data) {
 	print $f "import jdk.incubator.foreign.*;\n";
 	print $f "import java.lang.invoke.*;\n";
 
-	print $f "public class $lib->{name} {\n";
+	print $f formatCallback($c, $obj);
 
-	print $f " static ResourceScope scope() { return ResourceScope.globalScope(); }\n";
-	foreach $cname (@{$lib->{functions}}) {
-		my $c = $data{$cname};
-		my $tmp;
+	close $f;
+}
+
+# any struct remaining in toDump (but not in api)
+# FIXME: how to lookup obj?
+foreach $k (sort keys %{$toDump}) {
+	if ($k =~ m/struct:(.*)/) {
+		my $name = $1;
+		my $s = $data{$k};
+		my $path = nameToPath($output, "$package.$name");
+
+		die("missing struct $name") if !$s;
 
-		print $f " static final MethodHandle $c->{name}\$FH = Memory.downcall(\"$c->{name}\",\n";
-		$tmp = formatFunctionDescriptor($c);
-		print $f "$tmp);\n";
+		delete $toDump->{$k};
 
-		$tmp = formatFunction($c);
-		print $f "public static ";
-		print $f $tmp;
+		my $obj = findAPIObject($api, 'struct', $name);
+
+		open (my $f, ">", $path) || die ("Cannot open '$path' for writing");
+		exportStruct($f, $s, $obj);
+		close $f;
 	}
+}
 
-	print $f "}\n";
+# Dump enum types used by everything and not dumped elsehwere
+foreach $k (sort keys %{$toDump}) {
+	if ($k =~ m/enum:(.*)/) {
+		my $name = $1;
+		my $s = $data{$k};
+		my $path = nameToPath($output, "$package.$name");
 
-	close $f;
+		die("missing enum $name") if !$s;
+
+		open(my $f, ">", $path) || die ("Cannot open '$path' for writing");
+
+		exportEnum($f, $s);
+
+		close $f;
+	}
+}
+
+# Dump define types not dumped elsehwere
+foreach $k (sort keys %{$toDump}) {
+	if ($k =~ m/define:(.*)/) {
+		my $name = $1;
+		my $s = $data{$k};
+		my $path = nameToPath($output, "$package.$name");
+
+		die("missing define $name") if !$s;
+
+		open(my $f, ">", $path) || die ("Cannot open '$path' for writing");
+
+		print $f "package $package;\n" if $package;
+		print $f "public interface $s->{name} {\n";
+
+		foreach $m (@{$s->{values}}) {
+			# some parsing problem here
+			next if !$m->{value};
+
+			print $f " /**\n ($m->{comment}) */\n" if ($m->{comment});
+			print $f " public static final $defineType{$m->{type}} $m->{name} = $definePrefix{$m->{type}}$m->{value}$defineSuffix{$m->{type}};\n";
+		}
+
+		print $f "}\n";
+
+		close $f;
+	}
+}
+
+# and we're done
+exit 0;
+
+sub loadControlFile {
+	my $path = shift @_;
+	my %def = ();
+	my $target;
+
+	open (my $d,"<",$path);
+
+	while (<$d>) {
+		next if /\s*\#/;
+
+		chop;
+
+		if ($target) {
+			if (m/\s*\}\s*$/) {
+				undef $target;
+			} elsif (/^\s*(\S+)\s*(.*)/) {
+				my @options = split(/\s+/,$2);
+				push @{$target->{items}}, {
+					match => $1,
+					options => \@options
+				};
+			}
+		} elsif (/^(\w+)\s+(\S*)\s*(.*)\s+\{/) {
+			my @options = split(/\s+/,$3);
+
+			$target = {
+				type => $1,
+				name => $2,
+				options => \@options,
+				items => []
+			};
+			push @{$def{$1}}, $target;
+		} elsif (/\S/) {
+			die("invalid line: %_");
+		}
+	}
+
+	close $d;
+
+	return \%def;
 }
diff --git a/src/template/Memory.java b/src/template/Memory.java
index 1cdf9f6..023c828 100644
--- a/src/template/Memory.java
+++ b/src/template/Memory.java
@@ -181,7 +181,7 @@ public class Memory {
 			this.segment = segment;
 		}
 
-		public static ByteArray create(MemorySegment segment) {
+ 		public static ByteArray create(MemorySegment segment) {
 			return new ByteArray(segment);
 		}
 
@@ -242,16 +242,24 @@ public class Memory {
 	public static class ShortArray extends AbstractList<Short> implements Memory.Addressable {
 		final MemorySegment segment;
 
-		public ShortArray(MemorySegment segment) {
+		private ShortArray(MemorySegment segment) {
 			this.segment = segment;
 		}
 
-		public ShortArray(Frame frame, long size) {
-			this(frame.allocateArray(Memory.SHORT, size));
+ 		public static ShortArray create(MemorySegment segment) {
+			return new ShortArray(segment);
 		}
 
-		public ShortArray(Frame frame, short... values) {
-			this(frame.allocateArray(Memory.SHORT, values));
+		public static ShortArray createArray(MemoryAddress address, long length, ResourceScope scope) {
+			return create(MemorySegment.ofAddress(address, length, scope));
+		}
+
+		public static ShortArray createArray(long length, SegmentAllocator alloc) {
+			return create(alloc.allocateArray(Memory.SHORT, length));
+		}
+
+		public static ShortArray create(SegmentAllocator alloc, short... values) {
+			return create(alloc.allocateArray(Memory.SHORT, values));
 		}
 
 		public final MemoryAddress address() {
@@ -295,16 +303,24 @@ public class Memory {
 	public static class IntArray extends AbstractList<Integer> implements Memory.Addressable {
 		final MemorySegment segment;
 
-		public IntArray(MemorySegment segment) {
+		private IntArray(MemorySegment segment) {
 			this.segment = segment;
 		}
 
-		public IntArray(Frame frame, long size) {
-			this(frame.allocateArray(Memory.INT, size));
+		public static IntArray create(MemorySegment segment) {
+			return new IntArray(segment);
+		}
+
+		public static IntArray createArray(MemoryAddress address, long length, ResourceScope scope) {
+			return create(MemorySegment.ofAddress(address, length, scope));
+		}
+
+		public static IntArray createArray(long length, SegmentAllocator alloc) {
+			return create(alloc.allocateArray(Memory.INT, length));
 		}
 
-		public IntArray(Frame frame, int... values) {
-			this(frame.allocateArray(Memory.INT, values));
+		public static IntArray create(SegmentAllocator alloc, int... values) {
+			return create(alloc.allocateArray(Memory.INT, values));
 		}
 
 		public final MemoryAddress address() {
@@ -352,12 +368,20 @@ public class Memory {
 			this.segment = segment;
 		}
 
-		public LongArray(Frame frame, long size) {
-			this(frame.allocateArray(Memory.LONG, size));
+		public static LongArray create(MemorySegment segment) {
+			return new LongArray(segment);
 		}
 
-		public LongArray(Frame frame, long... values) {
-			this(frame.allocateArray(Memory.LONG, values));
+		public static LongArray createArray(MemoryAddress address, long length, ResourceScope scope) {
+			return create(MemorySegment.ofAddress(address, length, scope));
+		}
+
+		public static LongArray createArray(long length, SegmentAllocator alloc) {
+			return create(alloc.allocateArray(Memory.LONG, length));
+		}
+
+		public static LongArray create(SegmentAllocator alloc, long... values) {
+			return create(alloc.allocateArray(Memory.LONG, values));
 		}
 
 		public final MemoryAddress address() {
@@ -401,16 +425,24 @@ public class Memory {
 	public static class FloatArray extends AbstractList<Float> implements Memory.Addressable {
 		final MemorySegment segment;
 
-		public FloatArray(MemorySegment segment) {
+		private FloatArray(MemorySegment segment) {
 			this.segment = segment;
 		}
 
-		public FloatArray(Frame frame, long size) {
-			this(frame.allocateArray(Memory.FLOAT, size));
+ 		public static FloatArray create(MemorySegment segment) {
+			return new FloatArray(segment);
+		}
+
+		public static FloatArray createArray(MemoryAddress address, long length, ResourceScope scope) {
+			return create(MemorySegment.ofAddress(address, length, scope));
+		}
+
+		public static FloatArray createArray(long length, SegmentAllocator alloc) {
+			return create(alloc.allocateArray(Memory.FLOAT, length));
 		}
 
-		public FloatArray(Frame frame, float... values) {
-			this(frame.allocateArray(Memory.FLOAT, values));
+		public static FloatArray create(SegmentAllocator alloc, float... values) {
+			return create(alloc.allocateArray(Memory.FLOAT, values));
 		}
 
 		public final MemoryAddress address() {
@@ -454,16 +486,24 @@ public class Memory {
 	public static class DoubleArray extends AbstractList<Double> implements Memory.Addressable {
 		final MemorySegment segment;
 
-		public DoubleArray(MemorySegment segment) {
+		private DoubleArray(MemorySegment segment) {
 			this.segment = segment;
 		}
 
-		public DoubleArray(Frame frame, long size) {
-			this(frame.allocateArray(Memory.DOUBLE, size));
+ 		public static DoubleArray create(MemorySegment segment) {
+			return new DoubleArray(segment);
+		}
+
+		public static DoubleArray createArray(MemoryAddress address, long length, ResourceScope scope) {
+			return create(MemorySegment.ofAddress(address, length, scope));
 		}
 
-		public DoubleArray(Frame frame, double... values) {
-			this(frame.allocateArray(Memory.DOUBLE, values));
+		public static DoubleArray createArray(long length, SegmentAllocator alloc) {
+			return create(alloc.allocateArray(Memory.DOUBLE, length));
+		}
+
+		public static DoubleArray create(SegmentAllocator alloc, double... values) {
+			return create(alloc.allocateArray(Memory.DOUBLE, values));
 		}
 
 		public final MemoryAddress address() {
@@ -515,8 +555,16 @@ public class Memory {
 			return new PointerArray(segment);
 		}
 
-		public static PointerArray createArray(long size, SegmentAllocator alloc) {
-			return create(alloc.allocateArray(Memory.POINTER, size));
+		public static PointerArray createArray(MemoryAddress address, long length, ResourceScope scope) {
+			return create(MemorySegment.ofAddress(address, length, scope));
+		}
+
+		public static PointerArray createArray(long length, SegmentAllocator alloc) {
+			return create(alloc.allocateArray(Memory.POINTER, length));
+		}
+
+		public static PointerArray create(Frame alloc, MemoryAddress... values) {
+			return create(alloc.allocateArray(Memory.POINTER, values));
 		}
 
 		public final MemoryAddress address() {
@@ -557,23 +605,38 @@ public class Memory {
 		}
 	}
 
+	// This needs a separate scope from the array itself
 	public static class HandleArray<T extends Memory.Addressable> extends AbstractList<T> implements Memory.Addressable {
 		final MemorySegment segment;
+		final ResourceScope scope;
 		BiFunction<MemoryAddress,ResourceScope,T> create;
 
-		private HandleArray(MemorySegment segment, BiFunction<MemoryAddress,ResourceScope,T> create) {
+		private HandleArray(MemorySegment segment, BiFunction<MemoryAddress,ResourceScope,T> create, ResourceScope scope) {
 			this.segment = segment;
 			this.create = create;
+			this.scope = scope;
 		}
 
 		public static <T extends Memory.Addressable> HandleArray<T> create(MemorySegment segment, BiFunction<MemoryAddress,ResourceScope,T> create) {
-			return new HandleArray<>(segment, create);
+			return new HandleArray<>(segment, create, segment.scope());
+		}
+
+		public static <T extends Memory.Addressable> HandleArray<T> create(MemorySegment segment, BiFunction<MemoryAddress,ResourceScope,T> create, ResourceScope scope) {
+			return new HandleArray<>(segment, create, scope);
 		}
 
 		public static <T extends Memory.Addressable> HandleArray<T> createArray(long size, SegmentAllocator alloc, BiFunction<MemoryAddress,ResourceScope,T> create) {
 			return create(alloc.allocateArray(Memory.POINTER, size), create);
 		}
 
+		public static <T extends Memory.Addressable> HandleArray<T> createArray(long size, SegmentAllocator alloc, BiFunction<MemoryAddress,ResourceScope,T> create, ResourceScope scope) {
+			return create(alloc.allocateArray(Memory.POINTER, size), create, scope);
+		}
+
+		public static <T extends Memory.Addressable> HandleArray<T> createArray(MemoryAddress address, long size, BiFunction<MemoryAddress,ResourceScope,T> create, ResourceScope scope) {
+			return create(MemorySegment.ofAddress(address, size * Memory.POINTER.byteSize(), scope), create);
+		}
+
 		@Override
 		public final MemoryAddress address() {
 			return segment.address();
diff --git a/test-api/Makefile b/test-api/Makefile
index a458279..a7a8666 100644
--- a/test-api/Makefile
+++ b/test-api/Makefile
@@ -13,17 +13,23 @@ api_demo_SOURCES := $(wildcard src/api/test/*.java)
 all::
 	mkdir -p bin
 
-all:: bin/demo.built
+all:: bin/demo.built bin/libapi.so
 
 bin/api.built: bin/api.gen
 	$(JAVAC) $(JAVACFLAGS) -cp bin/classes -d bin/classes \
 		$(shell find bin/java -name '*.java')
 	touch $@
 
-bin/api.gen: bin/api.pm ../src/generate-native $(api_SOURCES)
-	../src/generate-native -d bin/java -t proto.api ./bin/api.pm
+bin/api.gen: bin/api.pm ../src/generate-native $(api_SOURCES)  bin/api-defines.pm api.api
+	../src/generate-native -v -d bin/java -t proto.api -a ./bin/api.pm -a ./bin/api-defines.pm api.api
 	touch $@
 
+bin/api-defines.pm: api.h ../src/export-defines api.api
+	../src/export-defines -d bin/api-defines.c api.api
+	$(CC) -o bin/api-defines -I. bin/api-defines.c
+	bin/api-defines $@~
+	mv $@~ $@
+
 bin/api.pm: api.h ../src/export.so
 	gcc -fplugin=../src/export.so -fplugin-arg-export-output=$@~ ./$< -o /dev/null
 	mv $@~ $@
diff --git a/test-api/README b/test-api/README
new file mode 100644
index 0000000..59a3438
--- /dev/null
+++ b/test-api/README
@@ -0,0 +1,8 @@
+
+Introduction
+------------
+
+Small generator and prototyping area for a c library.
+
+This is mostly to test out ideas like pattern matching for mostly
+automatic api conversion.
diff --git a/test-api/api.api b/test-api/api.api
new file mode 100644
index 0000000..cfe3103
--- /dev/null
+++ b/test-api/api.api
@@ -0,0 +1,15 @@
+
+# collect all functions and the defines we want
+library APILib load=api {
+	define:api-defines
+	func://
+}
+
+# grab all structures
+struct // default=all field:rename=studly-caps {
+}
+
+# grab all defines in api.h
+define api-defines api.h {
+	api.h 	    file-include
+}
diff --git a/test-api/api.c b/test-api/api.c
index 0aad6b8..58cac5f 100644
--- a/test-api/api.c
+++ b/test-api/api.c
@@ -1,6 +1,7 @@
 
 #include <stdio.h>
 #include <string.h>
+#include <stdlib.h>
 #include "api.h"
 
 static void funca(int a) {
diff --git a/test-api/api.h b/test-api/api.h
index e4639ca..4e6b023 100644
--- a/test-api/api.h
+++ b/test-api/api.h
@@ -1,4 +1,13 @@
 
+#define API_FOO "12"
+#define API_MAKEVERSION(a, b) (a <<16) | b
+#define API_VERSION API_MAKEVERSION(1, 0)
+#define API_A 12UL
+#define API_B 12L
+#define API_C 12U
+#define API_D 12.0
+#define API_E 12.0f
+
 struct data {
 	struct data *next;
 
@@ -22,3 +31,9 @@ struct api {
 
 void *api_func(const char *name);
 struct api *api_create(void);
+
+struct list_node {
+	struct list_node *succ;
+	struct list_node *pred;
+	char name[64-16];
+};
diff --git a/test-api/src/api/test/TestAPI.java b/test-api/src/api/test/TestAPI.java
index 2d8be3b..108d898 100644
--- a/test-api/src/api/test/TestAPI.java
+++ b/test-api/src/api/test/TestAPI.java
@@ -35,7 +35,7 @@ public class TestAPI {
 			b.setC((byte)255);
 			b.setD((byte)255);
 
-			b.setTest_a(cb);
+			b.setTestA(cb);
 
 			System.out.println("from a");
 			print_data(a);
@@ -49,7 +49,7 @@ public class TestAPI {
 
 			// dynamic lookup
 			System.out.println("call funca via symbol lookup");
-			Memory.FunctionPointer<Call_i32_v> funca = Call_i32_v.downcall(api_func(Memory.ByteArray.create("funca", frame)), scope);
+			Memory.FunctionPointer<Call_i32_v> funca = Call_i32_v.downcall(api_func("funca"), scope);
 			System.out.printf(" %s\n", funca.symbol());
 			funca.function().call(12);
 
diff --git a/test-ffmpeg/Makefile b/test-ffmpeg/Makefile
new file mode 100644
index 0000000..e1403fc
--- /dev/null
+++ b/test-ffmpeg/Makefile
@@ -0,0 +1,52 @@
+
+CFLAGS=-g -fPIC
+HOST_CC=gcc
+
+JAVA_HOME=/opt/jdk-foreign/jvm/openjdk-19-internal
+JAVAC=$(JAVA_HOME)/bin/javac
+JAVA=$(JAVA_HOME)/bin/java
+
+JAVACFLAGS=--add-modules jdk.incubator.foreign
+
+api_SOURCES := $(wildcard ../src/api/*.java)
+ffmpeg_SOURCES := $(wildcard src/proto/ffmpeg/*.java)
+ffmpeg_demo_SOURCES := $(wildcard src/ffmpeg/test/*.java)
+
+all::
+	mkdir -p bin
+
+all:: bin/demo.built
+
+bin/ffmpeg.built: bin/ffmpeg.gen $(ffmpeg_SOURCES)
+	$(JAVAC) $(JAVACFLAGS) -cp bin/classes -d bin/classes \
+		$(shell find bin/java -name '*.java') \
+		$(ffmpeg_SOURCES)
+	touch $@
+
+bin/ffmpeg.gen: bin/ffmpeg.pm bin/ffmpeg-defines.pm ../src/generate-native $(api_SOURCES)
+	../src/generate-native -d bin/java -t proto.ffmpeg -a ./bin/ffmpeg.pm -a ./bin/ffmpeg-defines.pm ffmpeg.api
+	touch $@
+
+bin/ffmpeg-defines.pm: ffmpeg.h ../src/export-defines ffmpeg.api
+	../src/export-defines -d bin/ffmpeg-defines.c ffmpeg.api
+	$(HOST_CC) -o bin/ffmpeg-defines -I. bin/ffmpeg-defines.c
+	bin/ffmpeg-defines $@~
+	mv $@~ $@
+
+bin/ffmpeg.pm: ffmpeg.h ../src/export.so
+	gcc -fplugin=../src/export.so -fplugin-arg-export-output=$@~ ./$< -o /dev/null
+	mv $@~ $@
+
+bin/demo.built: $(ffmpeg_demo_SOURCES) bin/ffmpeg.built
+	$(JAVAC) $(JAVACFLAGS) -cp bin/classes -d bin/classes $(ffmpeg_demo_SOURCES)
+	touch $@
+
+demo: all
+	$(JAVA)	--enable-native-access=ALL-UNNAMED --add-modules jdk.incubator.foreign \
+		-cp bin/classes \
+		ffmpeg.test.TestFFMPEG
+
+clean:
+	rm -rf bin 000[0-9].pnm
+
+.PHONY: demo clean all
diff --git a/test-ffmpeg/README b/test-ffmpeg/README
new file mode 100644
index 0000000..e22691f
--- /dev/null
+++ b/test-ffmpeg/README
@@ -0,0 +1,12 @@
+
+Introduction
+------------
+
+Small demonstrator of generate-native on a complex and messy api
+wrapped in a Java "object oriented" one.
+
+make demo
+---------
+
+Copy some video file to 'movie.avi' to run the demo.  It will output
+the first 10 frames to "\d{5}.pnm".
diff --git a/test-ffmpeg/ffmpeg.api b/test-ffmpeg/ffmpeg.api
new file mode 100644
index 0000000..777d287
--- /dev/null
+++ b/test-ffmpeg/ffmpeg.api
@@ -0,0 +1,222 @@
+
+# access=rwi
+#  r read (get)
+#  w write (set)
+#  i read/write indexed (get/setAtIndex)
+
+struct <default> field:rename=studly-caps func:rename=camel-case access=rw {
+}
+
+struct AVFormatContext default=none func:rename=s/^(avformat_|av_)//,camel-case field:rename=studly-caps {
+	iformat		access=r rename=InputFormat
+	oformat		access=r rename=OutputFormat
+	pb			 rename=IOContext
+
+	ctx_flags	access=r
+	nb_streams      access=r rename=NumStreams
+
+	streams		array-size=nb_streams
+
+	start_time      access=r
+	duration	access=r
+	bit_rate	access=r
+
+	interrupt_callback
+
+	func:avformat_open_input	static constructor=byref constructor-result=0 success=0
+	func:avformat_find_stream_info	rename=findStreamInfo instance=0
+
+	func:av_read_frame
+}
+
+struct AVStream default=none field:rename=studly-caps {
+	index		access=r
+	id		access=r rename=ID
+	time_base
+	start_time
+	duration
+	nb_frames	rename=NumFrames
+	discard
+	avg_frame_rate	rename=AverageFrameRate
+	sample_aspect_ratio
+	codecpar	rename=CodecParameters
+}
+
+struct AVCodec access=r default=none func:rename=s/^avcodec_//,camel-case field:rename=studly-caps {
+	id      		 rename=ID
+	name
+	long_name
+	type
+	capabilities
+	max_lowres
+	# need some sort of length=null-terminated here i suppose
+	supported_framerates	rename=framerates
+	pix_fmts			rename=pixelFormats
+
+	func:av_codec_next			static rename=next
+	func:avcodec_find_decoder		static
+	func:avcodec_find_decoder_by_name	static
+	# here or on AVCodecContext?
+	func:avcodec_alloc_context3		rename=allocContext
+
+	define:AVCodecBits
+}
+
+struct AVCodecContext default=none func:rename=s/^avcodec_//,camel-case field:rename=studly-caps {
+	codec_id      rename=CodecID
+
+	skip_loop_filter
+	skip_idct
+	skip_frame
+
+	func:avcodec_alloc_context3	static rename=alloc
+	func:avcodec_open2		rename=open
+	func:avcodec_send_packet
+	func:avcodec_receive_packet
+	func:avcodec_send_frame
+	func:avcodec_receive_frame
+
+	define:AVCodecContextBits
+}
+
+struct AVFrame default=all func:rename=s/^av_frame_//,camel-case field:rename=studly-caps {
+	func:av_frame_alloc  static
+	func:av_frame_free
+}
+
+struct AVFrameSideData default=none {
+}
+
+struct AVRational field:rename=studly-caps {
+}
+
+struct AVCodecParameters func:rename=s/^avcodec_parameters_//,camel-case field:rename=studly-caps {
+	codec_id		 rename=CodecID
+	func:avcodec_parameters_alloc		static
+	func:avcodec_parameters_free
+	func:avcodec_parameters_copy
+	func:avcodec_parameters_to_context	instance=1
+	func:avcodec_parameters_from_context
+}
+
+struct AVPacket default=all func:rename=s/^av_packet_//,camel-case field:rename=studly-caps {
+	pts|dts	   rename=upper-leadin
+	data	   array-size=size
+
+	func:av_packet_alloc  static
+	func:av_packet_free
+	func:av_init_packet   rename=init
+}
+
+struct SwsContext func:rename=s/^sws_// {
+#       func:/^sws_/
+	func:sws_getContext	static
+	func:sws_freeContext
+	func:sws_scale
+
+	define:sws
+}
+
+struct SwsFilter {
+}
+struct SwsVector {
+}
+struct AVPixFmtDescriptor {
+}
+struct AVComponentDescriptor access=rwi {
+}
+
+library AVUtil {
+	func:/^av_image_/
+}
+
+struct AVProbeData default=none field:rename=studly-caps {
+}
+
+struct AVBufferRef access=rwi default=none {
+}
+struct AVPacketSideData default=none {
+}
+struct AVIOContext default=none {
+}
+struct AVIOInterruptCB default=none {
+}
+
+struct AVDictionaryEntry {
+}
+
+struct AVDictionary func:rename=s/^av_dict_//,camel-case {
+	define:dict
+	func:/av_dict_/
+}
+
+struct AVInputFormat default=none access=r field:rename=studly-caps {
+	name
+	long_name
+	mime_type
+	flags
+	extensions
+# want to define functions here
+	func:av_iformat_next  	      static     rename=next
+	func:av_register_input_format instance=0 rename=register
+}
+
+struct AVOutputFormat default=none field:rename=studly-caps {
+	name
+	long_name
+	mime_type
+	flags
+	extensions
+	audio_codec
+	video_codec
+	subtitle_codec
+}
+
+define dict ffmpeg.h {
+	/AV_DICT_/
+}
+
+define AVChannelLayoutBits ffmpeg.h {
+	/^AV_CH_LAYOUT_/	x64
+}
+
+define AVErrorBits ffmpeg.h {
+	/^AVERROR_/		x32
+}
+
+define AVCodecContextBits ffmpeg.h {
+	/^AV_CODEC_FLAG_|AV_CODEC_FLAG2_/	u32
+	/^AV_INPUT_BUFFER_/	i32
+}
+
+define AVCodecBits ffmpeg.h {
+	/^AV_CODEC_CAP_/	x32
+}
+
+define AVIOContextBits ffmpeg.h {
+	/^AVSEEK_|AVIO_FLAG_|AVIO_SEEKABLE_/
+}
+
+define AVOptionsBits ffmpeg.h {
+	/^AV_OPT_/
+}
+
+define sws ffmpeg.h {
+	/^SWS_/
+}
+
+# This is used to define 'char *' arguments which are not strings.
+#  ... a bit clumsy
+
+func av_image_copy_plane {
+	src|dst		 array
+}
+func av_image_copy_to_buffer {
+	dst array
+}
+func av_image_fill_arrays {
+	src array
+}
+func av_image_fill_pointers {
+	ptr array
+}
diff --git a/test-ffmpeg/ffmpeg.h b/test-ffmpeg/ffmpeg.h
new file mode 100644
index 0000000..1d3a58e
--- /dev/null
+++ b/test-ffmpeg/ffmpeg.h
@@ -0,0 +1,6 @@
+
+#include <libavcodec/avcodec.h>
+#include <libavformat/avformat.h>
+#include <libswscale/swscale.h>
+#include <libavutil/imgutils.h>
+#include <libavutil/opt.h>
diff --git a/test-ffmpeg/src/ffmpeg/test/TestFFMPEG.java b/test-ffmpeg/src/ffmpeg/test/TestFFMPEG.java
new file mode 100644
index 0000000..cbec649
--- /dev/null
+++ b/test-ffmpeg/src/ffmpeg/test/TestFFMPEG.java
@@ -0,0 +1,137 @@
+
+package ffmpeg.test;
+
+import jdk.incubator.foreign.*;
+import java.io.FileOutputStream;
+import java.io.IOException;
+
+import proto.ffmpeg.Memory.*;
+import proto.ffmpeg.*;
+
+import static proto.ffmpeg.AVMediaType.*;
+import static proto.ffmpeg.AVPixelFormat.*;
+
+public class TestFFMPEG {
+
+	public static void main(String[] args) throws Exception {
+		System.loadLibrary("avcodec");
+		System.loadLibrary("avformat");
+		System.loadLibrary("swscale");
+		System.loadLibrary("avutil");
+
+		try (Frame frame = Memory.createFrame();
+			ResourceScope scope = ResourceScope.newConfinedScope()) {
+			int res;
+
+			if (false) {
+				AVInputFormat avif = null;
+				while ((avif = AVInputFormat.next(avif)) != null) {
+					System.out.printf("name: %30s %s\n",
+						avif.getName(),
+						avif.getLongName());
+				}
+			}
+
+			AVFormatContext format = AVFormatContext.openInput("movie.avi", null, null, scope);
+
+			if (format == null) {
+				throw new IOException("File not found: movie.avi");
+			}
+
+			format.findStreamInfo(null);
+
+			HandleArray<AVStream> streams = format.getStreams();
+
+			System.out.printf(" streams=%d %d\n", format.getNumStreams(), streams.size());
+
+			for (AVStream s: streams) {
+				AVCodecParameters cp = s.getCodecParameters();
+
+				System.out.printf(" stream [index=%d, id=%d]\n", s.getIndex(), s.getID());
+				System.out.printf("  start=%d\n  duration=%d\n", s.getStartTime(), s.getDuration());
+
+				System.out.printf("  codecType: %d\n", cp.getCodecType());
+				System.out.printf("  codecId: %d\n", cp.getCodecID());
+				System.out.printf("  bitrate: %d\n", cp.getBitRate());
+				System.out.printf("  samplerate: %d\n", cp.getSampleRate());
+			}
+
+			AVCodecParameters vcp = null;
+			AVStream vstream = null;
+
+			for (AVStream stream: streams) {
+				AVCodecParameters cp = stream.getCodecParameters();
+
+				switch (cp.getCodecType()) {
+				case AVMEDIA_TYPE_VIDEO:
+					System.out.printf("  video = %d\n", stream.getIndex());
+					vstream = stream;
+					vcp = cp;
+					break;
+				case AVMEDIA_TYPE_AUDIO:
+					System.out.printf("  audio = %d\n", stream.getIndex());
+					break;
+				}
+			}
+
+			AVCodec codec = AVCodec.findDecoder(vcp.getCodecID());
+			//AVCodecContext c = AVCodecContext.alloc(codec);
+			AVCodecContext c = codec.allocContext();
+			AVFrame avframe = AVFrame.alloc();
+			AVPacket packet = AVPacket.alloc();
+
+			packet.init();
+			vcp.toContext(c);
+			c.setCodecID(codec.getID());
+			res = c.open(codec, null);
+
+			System.out.printf("c.open = %d\n", res);
+
+			AVPixelReader pix = null;
+			long nframes = 0;
+			long pts = 0;
+
+			while ((res = format.readFrame(packet)) >= 0) {
+				int index = packet.getStreamIndex();
+
+				if (index == vstream.getIndex()) {
+					//boolean ready;
+					int ready;
+
+					pts = packet.getDTS();
+
+					res = c.sendPacket(packet);
+					res = c.receiveFrame(avframe);
+
+					if (res == 0) {
+						int w = avframe.getWidth();
+						int h = avframe.getHeight();
+						int fmt = avframe.getFormat();
+
+						if (true) {
+							if (pix == null) {
+								//pix = frame.getPixelReader(w, h, 0);
+								pix = new FramePixelReader(avframe, w, h, 0);
+							}
+
+							byte[] pixels = new byte[w*h*3];
+
+							pix.getPixels(0, h, AV_PIX_FMT_RGB24, pixels, 0, 0);
+
+							try (FileOutputStream fos = new FileOutputStream(String.format("%04d.pnm", nframes))) {
+								String header = String.format("P6\n%d %d\n255\n", w, h);
+								fos.write(header.getBytes());
+								fos.write(pixels);
+							}
+						}
+
+						System.out.printf("%06d video frame %dx%d @ %d\n", nframes, w, h, fmt);
+						nframes++;
+					}
+				}
+				if (nframes == 10)
+					break;
+			}
+		}
+	}
+}
diff --git a/test-ffmpeg/src/proto/ffmpeg/AVPixelReader.java b/test-ffmpeg/src/proto/ffmpeg/AVPixelReader.java
new file mode 100644
index 0000000..6ca4a3c
--- /dev/null
+++ b/test-ffmpeg/src/proto/ffmpeg/AVPixelReader.java
@@ -0,0 +1,53 @@
+/*
+ * Copyright (C) 2017 Michael Zucchi
+ *
+ * This file is part of jjmpeg <https://www.zedzone.space/software/jjmpeg.html>
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+package proto.ffmpeg;
+
+import java.nio.ByteBuffer;
+
+/**
+ * AVPixelReader is a high level interface to SwsContext.
+ * <p>
+ * This interface allows one to retrieve pixels by bands to a specific format.
+ *
+ * @see AVFrame#getPixelReader
+ */
+public interface AVPixelReader {
+
+	/**
+	 * Read pixels into a Buffer. The buffer must be a Direct*Buffer.
+	 *
+	 * @param <T>
+	 * @param y Start scanline. Currently only 0 works.
+	 * @param h End scanline output. Currently only the output height works.
+	 * @param fmt Pixel format to write to.
+	 * @param buffer Target for pixels.
+	 * @param scanlineStride Should match the output width and may be ignored.
+	 */
+	public void getPixels(int y, int h, int fmt, ByteBuffer buffer, int scanlineStride);
+
+	public void getPixels(int y, int h, int fmt, byte[] buffer, int offset, int scanlineStride);
+
+	public void getPixels(int y, int h, int fmt, short[] buffer, int offset, int scanlineStride);
+
+	public void getPixels(int y, int h, int fmt, int[] buffer, int offset, int scanlineStride);
+
+	public long bufferSize(int h, int fmt);
+
+	public void release();
+}
diff --git a/test-ffmpeg/src/proto/ffmpeg/FramePixelReader.java b/test-ffmpeg/src/proto/ffmpeg/FramePixelReader.java
new file mode 100644
index 0000000..26ec177
--- /dev/null
+++ b/test-ffmpeg/src/proto/ffmpeg/FramePixelReader.java
@@ -0,0 +1,153 @@
+/*
+ * Copyright (C) 2020 Michael Zucchi
+ *
+ * This file is part of jjmpeg <https://www.zedzone.space/software/jjmpeg.html>
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+package proto.ffmpeg;
+
+import java.nio.Buffer;
+import java.nio.ByteBuffer;
+import java.nio.BufferOverflowException;
+import jdk.incubator.foreign.*;
+import proto.ffmpeg.Memory.*;
+
+public class FramePixelReader implements AVPixelReader {
+
+	SwsContext sws;
+	final AVFrame frame;
+	final int dwidth, dheight, flags;
+	int dfmt = -1;
+
+	ResourceScope scope = ResourceScope.newImplicitScope();
+	MemorySegment seg;
+
+	public FramePixelReader(AVFrame frame, int dwidth, int dheight, int flags) {
+		this.frame = frame;
+		this.dwidth = dwidth;
+		this.dheight = dheight;
+		this.flags = flags;
+	}
+
+	SwsContext getContext(int fmt) {
+		if (sws == null || fmt != dfmt) {
+			dfmt = fmt;
+			if (sws != null) {
+				//sws.release();
+			}
+			sws = SwsContext.getContext(frame.getWidth(), frame.getHeight(), frame.getFormat(),
+				dwidth, dheight, dfmt, flags,
+				null, null, null);
+		}
+		return sws;
+	}
+
+	MemorySegment getSegment(long size) {
+		if (seg == null || seg.byteSize() < size) {
+			if (seg != null) {
+				//Memory.free(seg.baseAddress());
+			}
+			//seg = Memory.ofNative(Memory.malloc(size), size);
+			seg = MemorySegment.allocateNative(size, 16, scope);
+		}
+		return seg;
+	}
+
+	private void getPixels(int y, int h, int fmt, MemorySegment buffer, int scanlineStride) {
+		SwsContext ctx = getContext(fmt);
+
+		// FIXME: check sizes
+
+		try (Frame a = Memory.createFrame()) {
+			IntArray stride = IntArray.createArray(4, a);
+			PointerArray data = PointerArray.createArray(4, a);
+			ByteArray buf = ByteArray.create(buffer);
+			int res;
+
+			res = AVUtil.av_image_fill_linesizes(stride, fmt, dwidth);
+			res = AVUtil.av_image_fill_pointers(data, fmt, h, buf, stride);
+
+			System.out.printf(" base     %016x\n", buffer.address().toRawLongValue());
+			for (int i=0;i<4;i++) {
+				System.out.printf(" %d %6d %016x\n", i, stride.get(i), data.get(i).address().toRawLongValue());
+			}
+
+			if (res > 0 && res <= buffer.byteSize()) {
+				ctx.scale(
+					frame.getData(),
+					frame.getLinesize(),
+					y, h,
+					data, stride);
+			} else {
+				throw new BufferOverflowException();
+			}
+		}
+	}
+
+	public long bufferSize(int h, int fmt) {
+		try (Frame a = Memory.createFrame()) {
+			IntArray stride = IntArray.createArray(4, a);
+			PointerArray data = PointerArray.createArray(4, a);
+			int res;
+
+			res = AVUtil.av_image_fill_linesizes(stride, fmt, dwidth);
+			res = AVUtil.av_image_fill_pointers(data, fmt, h, null, stride);
+
+			return res;
+		}
+	}
+
+	public void getPixels(int y, int h, int fmt, ByteBuffer buffer, int scanlineStride) {
+		getPixels(y, h, fmt, MemorySegment.ofByteBuffer((ByteBuffer)buffer), scanlineStride);
+	}
+
+	public void getPixels(int y, int h, int fmt, byte[] buffer, int offset, int scanlineStride) {
+		SwsContext ctx = getContext(fmt);
+
+		try (Frame a = Memory.createFrame()) {
+			IntArray stride = IntArray.createArray(4, a);
+			PointerArray data = PointerArray.createArray(4, a);
+			MemorySegment seg = getSegment(bufferSize(h, fmt));
+			ByteArray dst = ByteArray.create(seg);
+			int res;
+
+			res = AVUtil.av_image_fill_linesizes(stride, fmt, dwidth);
+			res = AVUtil.av_image_fill_pointers(data, fmt, h, dst, stride);
+
+			AVUtil.av_image_fill_linesizes(stride, fmt, dwidth);
+
+			ctx.scale(
+				frame.getData(),
+				frame.getLinesize(),
+				y, h,
+				data, stride);
+
+			MemorySegment.ofArray(buffer).copyFrom(seg);
+		}
+	}
+
+	public void getPixels(int y, int h, int fmt, short[] buffer, int offset, int scanlineStride) {
+	}
+
+	public void getPixels(int y, int h, int fmt, int[] buffer, int offset, int scanlineStride) {
+	}
+
+	public void release() {
+		if (scope != null) {
+			scope.close();
+			scope = null;
+		}
+	}
+}