search

Compiler won't scale ﴾still﴿

 Moderated

jzakiya Mar 16

I raised this issue regarding 0.17.2 but it still persists in 0.18.0.

I want to generate some system constants at compile time.

I need to generate some constant arrays of numbers for a ﴾selectable﴿ prime generator ﴾PG﴿.

When the array size gets past some ﴾?﴿ value the compiler quits with this message:

generating parameters for P13

stack trace: (most recent call last)
twinprimes_ssozp13par5d.nim(81)
lib/system.nim(3806) genPGparameters
lib/system.nim(3806, 14) Error: interpretation requires too many iterations

This SERIOUSLY impedes my development process in Nim, which doesn't exist in C++.

I provide code for the case that will compile, then the version that the compiler squawks at.

The array residues for P13 ﴾prime generator for prime of 13﴿ contains 5760 values. The array restwins contains 2970
values.

When I attempt to compile array inverses , containing another 5760 values, the compiler squawks.

I ended up using Ruby to generate the inverses array values, then displayed to terminal, copied, pasted into an editor,
line formatted, copied, then pasted the formatted array values into my Nim source code so I could compile and run the
program.

I need to do the next higher prime generator P17, whose array residues|resinvrs each contain 92,160 values, but the
compiler ends when trying to create just the residues array.

So somewhere between ﴾5760 + 2970﴿ = 8730 ‐ 92,160 iterations, the compiler stops working.

Besides this being an arbitrary decision to have the compiler act like this, it's undocumented, and there is apparently no
﴾known﴿ mechanism to extend this arbitrary limit to a user. Is there some compiler flag to eliminate|extend it?

To say the least This makes me Very Unhappy!! :-( .

My development has stalled because of this. And it is totally not feasible for me to generate two 92,160 element arrays in
Ruby ﴾which it has no problem doing﴿ and go through the print|format|copy|paste dance of two 1000+ lines of numbers,
again. And that's just for P17. P19 has 1,658,880 values for each array,

It would make me very, very Happy :-) to eliminate this barrier to my continued project development.

# This code will compile

import math # for sqrt function

import strutils, typetraits # for number input
import times, os # for timing code execution
import osproc # for getting threads count
import threadpool # for parallel processing
{.experimental.} # required to use 'parallel' (0.17.x)

proc genPGparameters(prime: int): (int, int, int, seq[int], seq[int]) =

echo("generating parameters for P", prime)
let primes = [2, 3, 5, 7, 11, 13, 17, 19, 23]
var modpg = 1
for prm in primes: (modpg *= prm; if prm == prime: break)

var pc = 1
var residues: seq[int] = @[]
while pc < modpg: (pc += 2; if gcd(modpg, pc) == 1: residues.add(pc))
let rescnt = residues.len

var restwins: seq[int] = @[]

var j = 0
while j < rescnt-1:
if residues[j]+2 == residues[j+1]:
restwins.add residues[j]; restwins.add residues[j+1]; j.inc
j.inc
let rescntp = restwins.len

result = (modpg, rescnt, rescntp, residues, restwins)

# Generate at compile time the parameters for P13

const parameters = genPGparameters(13)
 # This won't compile with the addition of creating the 'inverses' array

import math # for sqrt function

import strutils, typetraits # for number input
import times, os # for timing code execution
import osproc # for getting threads count
import threadpool # for parallel processing
{.experimental.} # required to use 'parallel' (0.17.x)

proc genPGparameters(prime: int): (int, int, int, seq[int], seq[int], seq[int]) =

echo("generating parameters for P", prime)
let primes = [2, 3, 5, 7, 11, 13, 17, 19, 23]
var modpg = 1
for prm in primes: (modpg *= prm; if prm == prime: break)

var pc = 1
var residues: seq[int] = @[]
while pc < modpg: (pc += 2; if gcd(modpg, pc) == 1: residues.add(pc))
let rescnt = residues.len

var restwins: seq[int] = @[]

var j = 0
while j < rescnt-1:
if residues[j]+2 == residues[j+1]:
restwins.add residues[j]; restwins.add residues[j+1]; j.inc
j.inc
let rescntp = restwins.len

var inverses: seq[int] = @[]

j = 0
while j < rescnt:
let res = residues[j]
for inv in residues:
if res*inv mod modpg == 1: (inverses.add(inv); break)
j.inc

result = (modpg, rescnt, rescntp, residues, restwins, inverses)

# Generate at compile time the parameters for P13

const parameters = genPGparameters(13)

    Reply

StasB Mar 16

As a temporary workaround, couldn't you just take your existing Nim function, use it to generate your stuff at runtime,
then dump it into a .nim file?
  Reply

jzakiya Mar 16

I could use Ruby to create all the parameters and dump them into a file too, which would be easier.

I Don't want to jump through extra HOOPs because the Nim compiler is deficient!!

I don't have to do this in C++.

The compiler should work for the human, not the other way around.

PLEASE, PLEASE, PLEASE, FIX THE COMPILER!!

    Reply

StasB Mar 16

I could use Ruby to create all the parameters and dump them into a file too, which would be easier.

You could, couldn't you? It does sound better than doing this:

displayed to terminal, copied, pasted into an editor, line formatted, copied, then pasted the formatted array
values into my Nim source code

So maybe you should give it some consideration. Or you could wait for someone to add your desired compiler flag, I
suppose.

I don't have to do this in C++.

The compiler should work for the human, not the other way around.

I don't know who the C++ compiler works for, but it's certainly not human kind.

  Reply

Araq  Mar 16

1. It is perfectly feasible to write a Nim program that outputs an array literal as a string. The compiler supports this out of
 the box with staticExec and its caching mechanism. No need for Ruby.

2. You're the first person on earth that did run into this limit of Nim's VM, afaict anyway. Which exists to prevent endless
programs at compile‐time. There is currently no switch to extend it but the number of iterations is at 1.5 million
iterations. Edit compiler/vmdef.nim to increase the value and tell me at which value your programs start to compile.

3. It's a mystery to me in what sense C++ supports that out of the box but I never used its new constexpr features.

  Reply

StasB Mar 16

It's a mystery to me in what sense C++ supports that out of the box

C++ has supported the feature of throwing the compiler into an infinite loop at least since the introduction of an
accidentally Turing‐complete template system.

  Reply

xomachine Mar 16

I'm also reached this limit before, but avoided it by simplifying compile‐time code. You may try to find a way to reduce
number of function calls per loop iteration to fit the limit.
  Reply

Araq  Mar 16

I'm also reached this limit before, but avoided it by simplifying compile‐time code.

Alright, I take it back. :‐﴿

BTW let instead of const for computed lookup tables is also an option.

  Reply

jzakiya Mar 16

I started changing the MaxLoopIterations* = 1500_000 # max iterations of all loops line in compiler/vmdef.nim
in increments of 1 million, until it was 20M, recompiling after each change, and still got error. Then I went up in 100M
increments until 1 trillion, and still got error.

Do I need to recompile|rebuild Nim to make the change? Is merely changing the file sufficient? I doesn't seem so.

    Reply

jzakiya Mar 16

OK, I finally got it to compile with the original iteration count, but man, did I have to jump through hoops to do it. :‐﴾

To reduce the number of loops, instead of doing brute force trial-and-error to test each residue against the others to
find its inverse, I found the code here

https://rosettacode.org/wiki/Modular_inverse#Nim

on Rosetta Code, that had a Nim version already done. [Notice: the code there doesn't compile without changing proc
mulInv(a0, b0): int = to proc mulInv(a0, b0: int): int = . In fact, quite a few of those old Rosetta code Nim
examples don't compile now, but I digress.]

proc mulInv(a0, b0: int): int =

var (a, b, x0) = (a0, b0, 0)
result = 1
if b == 1: return
while a > 1:
let q = a div b
a = a mod b
swap a, b
result = result - q * x0
swap x0, result
if result < 0: result += b0

Then I was able to eliminate the loop‐in‐a‐loop construct, and just do this:

var inverses: seq[int] = @[]

for res in residues: inverses.add(mulInv(res,modpg))

I was trying to avoid using another function just to do this, since I don't use it at runtime, but you do what you gotta do. :‐﴾

Does Nim already have this function in its library? If not, can you please add it (math lib, etc).

Below is final code that compiles with original compiler iteration value, with some added diagnostic checks.

[Note: I changed function name to more standard|descriptive usage.]

 # Compute modular inverse of a to base b, e.g. a*a_inverse mod b = 1
proc modinv(a0, b0: int): int =
var (a, b, x0) = (a0, b0, 0)
result = 1
if b == 1: return
while a > 1:
let q = a div b
a = a mod b
swap a, b
result = result - q * x0
swap x0, result
if result < 0: result += b0

# Create constant parameters for chosen PG at compile time

proc genPGparameters(prime: int): (int, int, int, seq[int], seq[int], seq[int]) =
echo("generating parameters for P", prime)
let primes = [2, 3, 5, 7, 11, 13, 17, 19, 23]
var modpg = 1
for prm in primes: (modpg *= prm; if prm == prime: break)

var pc = 1
var residues: seq[int] = @[]
while pc < modpg: (pc += 2; if gcd(modpg, pc) == 1: residues.add(pc))
let rescnt = residues.len

var restwins: seq[int] = @[]

var j = 0
while j < rescnt-1:
if residues[j]+2 == residues[j+1]:
restwins.add residues[j]; restwins.add residues[j+1]; j += 1
j += 1
let rescntp = restwins.len

inverses: seq[int] = @[]

for res in residues: inverses.add(modinv(res,modpg))

echo("residues size = ", residues.len)

echo("inverses size = ", inverses.len)

for j in 0..residues.len-1: (if residues[j]*inverses[j] mod modpg != 1: echo("error at index ",j))

result = (modpg, rescnt, rescntp, residues, restwins, inverses)

# Generate at compile time the parameters for P13

const parameters = genPGparameters(13)

This must be is under the Magical Limit using P13. :‐﴿

So what will it do with P17? And the answer is.......Nope! It bombs ﴾again﴿ when trying to compute the 92,160 values for the
residues array, as it never gets to the inverse array computation. I ultimately tried changing the iteration variable upto 1
trillion again before giving up. I think I will need to rebuild Nim with a new value in that file to make it take. That's for
tomorrow ﴾maybe﴿, unless someone can instruct me how to do it without rebuilding Nim.

So, at least partial success, and I have learned something new and useful.

But Please, FIX THIS!, and DOCUMENT IT!

    Reply

jzakiya Mar 16

One ﴾last?﴿ trick to reduce the number of iterations.

I changed this:

var residues: seq[int] = @[]

var pc = 1
while pc < modpg: (pc += 2; if gcd(modpg, pc) == 1: residues.add(pc))
let rescnt = residues.len

to this:

var residues: seq[int] = @[]

var pc = 1
var inc = 4
while pc < modpg: (pc += inc; inc = inc xor 0b110; if gcd(modpg, pc) == 1: residues.add(pc))
let rescnt = residues.len

This reduces the number of times gcd is called from half the numbers ﴾all the odds﴿ upto modpg ﴾modpg/2﴿ to a third
﴾modpg/3﴿.

This change produces the P3 PG residue sequence: pcs = 6m +|- 1

 pc = 1, 5, 7, 11, 13, 17, 19...
+4 +2 +4 +2 +4 +2

So while it shaves off some iterations of gcd it's still not enough to get P17 to compile.

A REQUEST: It would be really, really nice to use the boolean operators like: x ^= y; x &= y ; etc.

    Reply

jzakiya Mar 16

Compiling again for P17, with 0.18.0 and 0.17.2, the error messages point to gcd function in both cases:

# 0.18.0 error message

generating parameters for P17

stack trace: (most recent call last)
twinprimes_ssozp17par5d3.nim(87)
twinprimes_ssozp17par5d3.nim(65) genPGparameters
lib/pure/math.nim(452) gcd
lib/pure/math.nim(452, 15) Error: interpretation requires too many iteratio

# 0.17.2 error message

generating parameters for P17

stack trace: (most recent call last)
twinprimes_ssozp17par5d3.nim(87)
twinprimes_ssozp17par5d3.nim(65) genPGparameters
lib/pure/math.nim(439) gcd
lib/pure/math.nim(439, 15) Error: interpretation requires too many iterations

In both cases it points to the same line in lib/pure/math.nim ﴾line 452 in 0.18.0 or 439 in 0.17.2﴿:

proc gcd*[T](x, y: T): T =

## Computes the greatest common divisor of ``x`` and ``y``.
## Note that for floats, the result cannot always be interpreted as
## "greatest decimal `z` such that ``z*N == x and z*M == y``
## where N and M are positive integers."
var (x,y) = (x,y) # <---------------- this line is pointed to in both cases
while y != 0:
x = x mod y
swap x, y
abs x

Changing var (x,y) = (x,y) to var x = x; var y = y or changing variable names, in both cases, creates same results.

Why does this line cause this compiler iteration problem?

    Reply

Araq  Mar 17

Do I need to recompile|rebuild Nim to make the change?

Yup. koch boot -d:release creates the modified compiler if you have an installation where koch boot works ﴾pick the
"installation from source" route: https://github.com/nim‐lang/Nim#compiling

Why does this line cause this compiler iteration problem?

Because that line triggered the 1_500_000th iteration limit. This line is nothing special, any line could trigger this limit.

  Reply

StasB Mar 17

Why not add a flag to set/disable the iteration limit like he wants? I'm sure that as the user base grows, more people are
going to hit the limit and get annoyed.
  Reply

Araq  Mar 17

Why not add a flag to set/disable the iteration limit like he wants? I'm sure that as the user base grows, more
people are going to hit the limit and get annoyed.

Because I am curious whether it can be supported out of the box by a "reasonable" limit first.

  Reply

StasB Mar 17
Such a limit would probably be around a few seconds, rather than a fixed number of iterations, because that's where most
people start to suspect that their computation is either too long or stuck in an infinite loop.
  Reply

Araq  Mar 17

Then the limit would depend on the used CPU. Bad idea IMO.
  Reply

StasB Mar 17

You're right. It would cause code that compiles on one machine to fail on another. Still, if the goal was to prevent infinite
loops, then a limit which causes the computation to fail almost immediately ﴾time‐wise﴿ on most machines makes little
sense, so even if it's fixed, it should at least be tuned to match a long‐ish computation on modern machines.
  Reply

jzakiya Mar 17

OK, I set MaxLoopIterations* = 1_000_000_000 ﴾1 Billion﴿ in ~/nim-0.18.0/compiler/vmdef.nim and rebuilt doing

./koch boot -d:release and was able to compile P17.

I think this value is way more reasonable, because as stated previously, on modern machines ﴾or really any system with
gigahertz clock speeds﴿ we're only talking seconds for even a billion iterations.

I really hope you will up the value. Until so, I'll patch my own systems. Thanks for instructing how to do this.

Again, I strongly urge you to PROMINENTLY DOCUMENT this capability too.

    Reply

jyapayne Mar 17

Araq, maybe it would be more appropriate to have this as a warning instead of stopping compilation completely?
Something like "Warning: Compiler has been through 1_000_000_000 iterations, compile time code might be in an infinite
loop"
  Reply

Araq  Mar 18

@jyapayne : Not sure, I like it to error out. Warning&continue doesn't seem right for what is ultimately a batch process.
@jzakiya : The limit is at one billion now.

  Reply

jzakiya Mar 19

@jzakiya : The limit is at one billion now.

+1!! :‐﴿

    Reply

4 MONTHS SINCE LAST REPLY

 Reply