MAX TNT Log Messages

This chapter covers the following topics:

Fatal and warning error messages

Definitions of fatal errors

Definitions of warning messages

Fatal crash information on console

Syslog messages

Fatal and warning error messages
Definitions of fatal errors
Definitions of warning messages
Fatal crash information on console
Syslog messages

The MAX TNT logs fatal and warning error messages to the fatal error log. If the system crashes before creating a log entry, it prints a stack trace to the console serial port. System-status messages, however, go to the Syslog host (if enabled) and the Status log.

Fatal and warning error messages

Each time the MAX TNT reboots, it logs a fatal error message to the fatal error log. The fatal error log also notes Warnings, which indicate situations that did not cause the MAX TNT to reset. Development engineers use Warnings for troubleshooting purposes. When a Warning occurs, the MAX TNT has detected an error condition and has recovered from it. Available flash space limits the number of entries in the fatal error log , and entries rotate on a First-in, First-out (FIFO) basis. You can clear the log by using the Clr-History command.

Format of fatal and warning error messages

Fatal and warning messages have the format shown in the following example:

WARNING:  Index: 171  Revision: 2.0.0 Slot 9/2 (tnthdlc)

Date: 02/28/1998.       Time: 20:57:59

Location: e0020b54 e006f568 e005d6b8 e005fd90 e005e4dc e00770a8

The first line indicates the type of error (fatal or warning), the index number of the error, the software revision number, the shelf and slot on which the error occurred, and the software load. The fatal log from older software versions display shelf 0.

The second line shows the date and time of the error.

The third line displays the top six program counter addresses from the execution stack active at the time of the crash.

Definitions of fatal errors

Following are definitions, by index number, of the fatal errors that the MAX TNT can report: if you experience a fatal error, contact Ascend Technical Support.


Index	Definition
1	Assert invoked during program execution. An Assert has been placed in the code. This problem can be either hardware related or software related.
2	Out of memory during memory allocation This is an out-of-memory condition, sometimes termed a memory leak.
3	Bad profile (T1 DSL related)
4	Switch type bad
5	LIF error
6	LCD error
7	ISAC (BRI) timeout BRI physical layer timeout.
8	Processor exception A processor-exception error caused the reset.
9	Invalid task switch (EXEC)
10	No mail descriptor (EXEC) This reset occurs if the MAX TNT tries to allocate a mail message when there are none left. The cause is usually a memory leak.
11	No mail buffer memory (EXEC)
12	No task to run (EXEC)
13	No timer memory (EXEC)
14	No timer pool (EXEC)
15	Wait called while in critical section (EXEC)
16	DSP not responding
17	DSP protocol error
18	DSP internal error
19	DSP loss of sync
20	DSP unused
21	DDD not responding
22	DDD protocol error
23	X25 buffer error
24	X25 init error
25	X25 stack error
27	Memory allocation of zero length
28	Memory allocation of negative length
29	Task infinite loop The reset was the result of a software loop.
30	Too large memory copy
31	Magic sequence missing (MEMCPY)
32	Wrong magic sequence (MEMCPY)
33	Bad start address (MEMCPY)
34	IDEC timeout IDSL physical layer timeout.
35	EXEC restricted
36	Stack overflow
37	DRAM card error Indicates that a DRAM card of unknown size is inserted in the DRAM slot or that the DRAM card failed POST. Applies to the Pipeline 220 only.
39	No priority 2 task This error occurs if the MAX TNT has not run a priority 2 task in the last minute. Tasks in the MAX TNT are assigned priorities. Because the main routing task runs at priority 2, this error means that the MAX TNT has been hung for 1 minute.
40	Protection fault
99	Operator reset This reset is logged immediately before the MAX TNT goes down. Instead of a standard stack backtrace, the message includes the active security-profile index. 0 (zero) indicates an unknown security profile. On the MAX TNT, the Default profile is number 1, and the Full Access profile is number 9.
100	System up As a complement to entry 99, this entry is logged as the MAX TNT is coming up. For a normal, manual reset, you should see a fatal error 99 followed by a fatal error 100.

Definitions of warning messages

Warnings are not the results of reset conditions. Most are detected problems from which the MAX TNT typically recovers fully. Following are the definitions, by index number, of the warnings the MAX TNT can report. Warning messages, by themselves, are not necessarily cause for concern. They are used by development engineers to determine the cause of fatal errors. Contact technical support if warning messages are accompanied by fatal errors.


Index	Definition
101	Buffer already in use
102	Buffer belongs to wrong pool
103	Buffer belongs to wrong heap
104	Buffer not previously allocated This warning can be logged under different conditions. For example, double freeing of memory and low-memory conditions can both generate a warning 104.
105	Buffer bad memory allocation
106	Buffer belongs to bogus pool
107	Buffer belongs to bogus heap Memory management code (or other modules) detected that the buffer header of what should have been a free buffer was corrupted by the previous overwrite.
108	Buffer negative length memory allocation A negative length request was made to the memory allocation code.
109	Buffer zero length memory allocation This warning is similar to Warning 108, except that a zero length request is made to the memory allocation code.
110	Error in buffer boundary
111	Error buffer too big Indicates that a software routine has tried to allocate a block of memory greater than 64Kbytes.
112	Error buffer null
113	Error buffer segment count zero
114	Error buffer trailer magic
115	Error in buffer trailer
116	Error in buffer trailer length
117	Error in buffer trailer user magic
118	Error buffer write after free
119	Error buffer not in use
120	Error buffer magic in memory copy
121	Error next buffer magic in memory copy
129	Error buffer no extra DRAM detected. Pipeline 75 version 2 only. No extra DRAM detected, or the DRAM failed the memory test.
130	PPP async buffer in use Indicates a PPP error.
140	Error no timers
145	LCD memory allocation failure Indicates that a memory-copy routine was called, but the source buffer was much larger than expected.
150	Error memory copy too large
151	Error memory copy magic missing
152	Error memory copy wrong magic
153	Error memory copy bad start address
154	WAN buffer leak Indicates an error in the WAN drivers.
160	Error in terminal-server state Indicates an error in the WAN drivers.
161	Error in terminal server semaphore
165	Error in telnet free driver
170	STAC timeout Indicates a hardware error in the STAC compression chip.
171	STAC data not owned Error in the STAC compression chip.
175	EXEC failure Indicates that there is insufficient memory to start a new task.
176	EXEC restricted
177	EXEC no mailbox
178	EXEC no resources
179	Unexpected error
180	Channel map stuck Caused by a missing channel on a T1/PRI line.
181	Channel display stuck
182	New call without disconnect request Indicates that a Disconnect message to the Central Office (CO) was not sent. The problem can be caused by conditions on the MAX TNT or at the CO. When the MAX TNT encounters the condition, it assumes the CO is correct, and answers the call.
183	New call without disconnect response
184	Disconnect request dropped
185	Spyder buffer error
186	Spyder descriptor error
190	TCP send buffer too big
191	TCP sequence gap
192	TCP too much data
193	TCP write attempt too large
194	TCP options bad
195	Modem message parsing failed
301	TACACS Plus pointer inconsistency
302	TACACS Plus index inconsistency
303	TACACS Plus TCP inconsistency
304	TACACS Plus TCP out-of-range socket
305	TACACS Plus socket mismatch
306	TACACS Plus unexpected authentication state
381	Error in filter list
382	Error no count in filter list
383	Error mismatch count filter list
550	No Ethernet transmit buffer
1001	Waiting for Ethernet controller
1002	Ethernet ACK command failed
1003	Ethernet reset invoked
1006	Ethernet controller unavailable (wait fail)
1010	Bad Ethernet transmit interrupt
1011	Ethernet transmit not completed

Fatal crash information on console

If the MAX TNT crashes without being able to write to the fatal error log, it prints a stack trace to the console serial port at the bit rate defined in the Serial profile. The trace reports the following information:

FE: N, Load: loadname, Version: version

Stack trace: 0xaddr-0 0xaddr-1 0xaddr-2 0xaddr-3 0xaddr-4 0xaddr-5

The first line indicates the number of the error and the software revision number.

The second line displays the top six program counter addresses from the execution stack active at the time of the crash.

Syslog messages

Syslog offloads to a host computer, known as the Syslog host. The Host parameter in the Log profile specifies the Syslog host, which saves the system status messages in a log file.

See the UNIX man pages about logger(1), syslog(3), syslog.conf(5), and syslogd(8) for details of the syslog daemon. The Syslog function requires UDP port 514.

The MAX TNT can report the following session data about various errors logged via Syslog:

Data

Description

[shelf/slot/line/channel]

Physical channel identifier.

[MBID xxx]

Session identifier.

[name]

The authenticated name.

[ calling -> called ]

The calling number or the called number, or both.

Progress code

An Ascend-specific code indicating the progress of the call. (For a list of progress codes, see the MAX TNT Reference Guide.)

Disconnect code

An Ascend-specific code indicating the reason the call was disconnected. (For a list of disconnect codes, see the MAX TNT Reference Guide.)

Data	Description
`[shelf/slot/line/channel]`	Physical channel identifier.
`[MBID` xxx`]`	Session identifier.
`[name]`	The authenticated name.
`[ calling -> called ]`	The calling number or the called number, or both.
`Progress code`	An Ascend-specific code indicating the progress of the call. (For a list of progress codes, see the MAX TNT Reference Guide.)
`Disconnect code`	An Ascend-specific code indicating the reason the call was disconnected. (For a list of disconnect codes, see the MAX TNT Reference Guide.)

For a given session identifier, multiple physical channel identifiers are possible. For example, one identifier might be for the T1 line, and another for the HDLC channel or modem number. This is shown in the sample log below, in which messages include the MBID, DNIS, and CLID in brackets. In this example, slot 1/2 is an 8T1 card, and slot 1/3 is a 48-modem card.

...: [1/2/1/2] [MBID 1; 9995551212 -> 7898] Incoming Call

...: [1/3/1/0] [MBID 1; 9995551212 -> 7898] Assigned to port

...: [1/2/1/2] [MBID 1; 9995551212 -> 7898] Call Connected

...: [1/3/1/0] [MBID 1] [johnc-pc] LAN session up: <johnc-pc>

...: [1/3/1/0] [MBID 1] [johnc-pc] LAN session down: <johnc-pc>

...: [1/3/1/0] [MBID 1; 9995551212 -> 7898] Call Terminated

...: [1/3/1/0] [MBID 1] [johnc-pc] : STOP: 'johnc-pc'; cause 45.; 
progress 60.; host 10.1.26.2

End of call information

If the Call-Info parameter is set to End-of-Call, the MAX TNT reports the following information to Syslog at the end of each authenticated call:

Station name
Calling phone number
Called phone number
Encapsulation protocol
Data rate (in bits per second)
Progress code and disconnect reason
Number of seconds before authentication
Number of bytes or packets received during authentication
Number of bytes or packets sent during authentication
Length of session (in seconds)
Number of bytes or packets received during the session
Number of bytes or packets sent during the session

The following example of a Syslog message shows the information it provides about the terminated call:

"Conn=("cjones-p50" 5106785291->? PPP 56000 60/185) \

Auth=(3 347/12 332/13) \

Sess=(1 643/18 644/19), Terminated"

The information also appears in the connection-status window, and is logged as a message at level Info.

If some of the information is not available, that field displays either a question mark (for strings) or a zero (for numerals).

DNIS and CLID information

Syslog messages pertaining to a call display DNIS and CLID information, provided that the information is known. Following is an example that shows the DNIS 7895 in Syslog messages:

LOG info, Shelf 1, Controller, Time: 17:48:56--

 shelf 1, slot 1, line 1, channel 6, dnis 7895, Incoming Call, MBID 
001

LOG info, Shelf 1, Controller, Time: 17:48:56--

 shelf 1, slot 2, dnis 7895, Assigned to port, MBID 001

LOG info, Shelf 1, Controller, Time: 17:48:57--

 shelf 1, slot 1, line 1, channel 6, dnis 7895, Call Connected, MBID 
001

LOG warning, Shelf 1, Controller, Time: 17:49:20--

 shelf 1, slot 1, line 1, channel 6, dnis 7895, Call Disconnected

LOG info, Shelf 1, Controller, Time: 17:49:20--

 shelf 1, slot 2, Call Terminated

Syslog messages initiated by a Secure Access Firewall

Depending on the settings specified in Secure Access Manager (SAM), the MAX TNT might generate Syslog packets about packets detected by Secure Access Firewall. By default, SAM specifies generation of a Syslog message about every packet blocked by the firewall. All messages initiated by a firewall are in the following format:

date time router name ASCEND: interface message

date is the date the message was logged by syslog.
time is the time the message was logged by syslog.
router is the router this message was sent from.
interface is the name of the interface (ie0, wan0, and so on), unless a call filter logs the packet as it brings up the link, in which case the word call appears.
The message format has a number of fields, one or more of which may be present.

The message fields appear in the following order:

protocol local direction remote length frag log tag

Table B-1. Syslog message fields for Secure Access Firewalls

Field

Description

protocol

Can be the four hexadecimal character Ether Type or one of the following network protocol names: ARP, RARP, IPX, Appletalk. For IP protocols, the field contains either the IP protocol number (up to 3 decimal digits) or one of the following names: IP-in-IP, TCP, ICMP, UDP, ESP, AH. In the special case of ICMP, the field also includes the ICMP Code and Type ([Code]/[Type]/icmp).

local

For non-IP packets, local is the source Ethernet MAC address of transmitted packets and the destination Ethernet MAC address of received packets. For a nonbridged WAN connection, the two MAC addresses are zeros. For IP protocols, local is the IP source address of transmitted packets and the IP destination address of received packets. In the case of TCP or UDP, it also includes the TCP or UDP port number ([IP-address];[port]).

direction

An arrow (<- or ->) indicating the direction in which the packet was traveling (receive and send, respectively).

remote

For non-IP protocols, remote has the same format that local has non-IP packets, but remote shows the destination Ethernet MAC address of transmitted packets and the source Ethernet MAC address of received packets. For IP protocols, remote has the same format as local but shows the IP destination address of transmitted packets and the IP source address of received packets.

length

The length of the packet in octets (8-bit bytes).

frag

Indicates that the packet has a nonzero IP offset or that the IP More-Fragments bit is set in the IP header.

log

Reports one or more messages based upon the packet status or packet header flags. The packet status messages include:
corrupt-the packet is internally inconsistent
unreach-the packet was generated by an "unreach=" rule in the firewall
!pass-the packet was blocked by the data firewall
bringup-the packet matches the call firewall
!bringup-the packet did not match the call firewall
TCP flag bits that will be displayed include syn, fin, rst.
syn is will only be displayed for the initial packet which has the SYN flag and not the ACK flag set.

tag

contains any user defined tags specified in the filter template used by SAM.

Table B-1. Syslog message fields for Secure Access Firewalls
Field	Description
protocol	Can be the four hexadecimal character Ether Type or one of the following network protocol names: ARP, RARP, IPX, Appletalk. For IP protocols, the field contains either the IP protocol number (up to 3 decimal digits) or one of the following names: IP-in-IP, TCP, ICMP, UDP, ESP, AH. In the special case of ICMP, the field also includes the ICMP Code and Type ([Code]/[Type]/icmp).
local	For non-IP packets, local is the source Ethernet MAC address of transmitted packets and the destination Ethernet MAC address of received packets. For a nonbridged WAN connection, the two MAC addresses are zeros. For IP protocols, l`ocal` is the IP source address of transmitted packets and the IP destination address of received packets. In the case of TCP or UDP, it also includes the TCP or UDP port number ([IP-address];[port]).
direction	An arrow (<- or ->) indicating the direction in which the packet was traveling (receive and send, respectively).
remote	For non-IP protocols, remote has the same format that local has non-IP packets, but remote shows the destination Ethernet MAC address of transmitted packets and the source Ethernet MAC address of received packets. For IP protocols, remote has the same format as local but shows the IP destination address of transmitted packets and the IP source address of received packets.
length	The length of the packet in octets (8-bit bytes).
frag	Indicates that the packet has a nonzero IP offset or that the IP More-Fragments bit is set in the IP header.
log	Reports one or more messages based upon the packet status or packet header flags. The packet status messages include: corrupt-the packet is internally inconsistent unreach-the packet was generated by an "unreach=" rule in the firewall !pass-the packet was blocked by the data firewall bringup-the packet matches the call firewall !bringup-the packet did not match the call firewall TCP flag bits that will be displayed include syn, fin, rst. syn is will only be displayed for the initial packet which has the SYN flag and not the ACK flag set.
tag	contains any user defined tags specified in the filter template used by SAM.

The backoff queue error message in the Syslog file

Accounting records are kept until they are acknowledged by the accounting server. Up to 100 unacknowledged records are stored in the backoff queue. If the unit never receives an acknowledgment to an accounting request, it will eventually run out of memory. In order to keep this situation from the occurring, the unit deletes the accounting records and displays this error message in the syslog file:

Backoff Q full, discarding user username

This error generally occurs for one of the following reasons:

You enabled RADIUS accounting on the MAX TNT, but not on the RADIUS server.
The Acct-Port or Acct-Key are incorrect. The Acct-Key must match the value assigned in the RADIUS clients file or the TACACS+ configuration file.
You are using the Livingston server instead of the Ascend server.

techpubs@eng.ascend.com