3702I stuck in "Initiated" on Mobility Express 8.10.196.0: "Downloading 5" counter stuck, AP asked to download image "to 0.0.0.0"
Hi guys, recently in company we bought 2 refurnished AIR-CAP3702I-E-K9. I've been fighting this for a couple of weeks and could use a second pair of eyes.
Setup
- Mobility Express 8.10.196. 0 running on an AIR-AP1832I-E-K9
- 4x AIR-CAP3702I-E-K9 joined and working fine (primary 8.10.196.0, backup 8.10.185.0)
- APs on trunk ports, native VLAN 5 (management), DHCP for VLAN 5 on the core switch
- No Cisco support contract, so I can't just download images whenever I need them
Background: what happened before
I bought two refurbished 3702Is from a local supplier. Both failed:
- AP #1 started dropping off the controller (Echo Timer Expiry, DTLS closed, constant image download attempts). The WLC log was full of
IMAGE_DOWNLOAD_ERR2: Refusing image download request ... max downloads (5) in progress. After a power cycle it wouldn't boot at all: "no bootable files". In ROMMON,ap3g2-k9w8-xx.153-3.JK11(the actual IOS file) was 0 bytes, and it was the only image on flash. No RCV image. - AP #2 (the spare) boot-looped with
%DOT11-2-FAILURE_RADIO_RESET ... hostmem badmagicand "Radio FW image download failed" on both radios. It came up on maybe every 4th boot. Clearly hardware.
Before sending them back, I caught AP #2 in a good boot and used archive tar /create to pull its JK11 image to my TFTP server (Ubuntu + tftpd-hpa). The resulting tar has no folder prefix, so I repacked it so the paths start with ap3g2-k9w8-mx.153-3.JK11/. I also pulled an RCV image off another AP the same way. So now I at least have backups of both.
Where I am now
The supplier sent a replacement 3702I. It came with JD4/JF5 and an old RCV (15.3(3)JD). I cleaned up flash, loaded JK11 from ROMMON with tar -xtract, set BOOT to JK11 with RCV as fallback, and it boots fine:
Cisco IOS Software, C3700 Software (AP3G2-K9W8-M), Version 15.3(3)JK11
LWAPP image version 8.10.196.0
AP image integrity check PASSED
So the version matches the controller exactly. The AP gets DHCP, finds the WLC, DTLS comes up, and it even shows in show ap summary and the ME dashboard. But it never actually comes up properly. On join, the WLC tells it to download an image that doesn't exist:
%CAPWAP-5-SENDJOIN: sending Join Request to 172.16.5.200
perform archive download capwap:/c3700 tar file
%CAPWAP-6-AP_IMG_DWNLD: Required image not found on AP. Downloading image from Controller.
%Error opening capwap:/c3700 (OK)
Download image failed, notify controller!!! From:8.10.196.0 to 0.0.0.0, FailureCode:4
capwap_image_proc: unable to open tar file
After that, Join Requests go unanswered and the WLC closes DTLS every ~60 seconds. Repeat forever.
WLC side
%CAPWAP-4-INVALID_STATE_EVENT: ... AP(4c:77:6d:5d:ea:f0) event (Capwap_join_request) and state (Capwap_image_data) combination
%LWAPP-3-IMAGE_DOWNLOAD_ERR6: AP did not complete Pre-Download Image 4c:77:6d:5d:ea:f0 - During scheduled autoreboot
%LWAPP-3-IMAGE_DOWNLOAD_ERR7: Waiting for 1 APs to complete their Pre-Download Image - For scheduled autoreboot.
Earlier, while the AP was still on its factory RCV (8.3.102.0):
%CAPWAP-3-DISC_MAX_DOWNLOAD: Ignoring discovery request from AP 4c:77:6d:43:dd:64 - maximum number of downloads (5) exceeded
show ap image all:
Initiated....... 1
Downloading..... 5
AP Name Primary Backup Predownload Status
AP-SPRAT-01 8.10.196.0 8.10.185.0 None
AP-SPRAT-02 8.10.196.0 8.10.185.0 None
AP-PRIZ-02 8.10.196.0 8.10.185.0 None
AP-PRIZ-01 8.10.196.0 8.10.185.0 None
AP-HALA-01 8.10.196.0 8.10.185.0 None
AP4c77.6d43.dd64 8.10.196.0 0.0.0.0 Initiated
The counter says 5 downloading, but no AP is actually downloading. I suspect these slots have been stuck since a previous upgrade (8.10.185.0 → 8.10.196.0), and that this is also what killed AP #1 in the first place.
What I've tried
show reset: no reset scheduledconfig ap image predownload abort all: accepted, no changeconfig ap image predownload abort <AP name>: accepted, no change
What would you do next? I would really appreciate your help. I can post the logs of AP and WLC if it would be helpful.
