码迷,mamicode.com
首页 > 其他好文 > 详细

neutron dhcp 高可用

时间:2020-07-22 20:51:02      阅读:75      评论:0      收藏:0      [点我收藏+]

标签:word   root   compute   default   mes   out   停止   执行   节点   

 

neutron agent-list | grep dhcp-agent
| 60c33247-ffcd-4a0a-bb2f-75535ff56f75 | DHCP agent           | controller3        | nova              | :-)   | True           | neutron-dhcp-agent        |
| 84a54e6d-f7d5-4f25-910b-5d025aa6aa3b | DHCP agent           | controller2        | nova              | :-)   | True           | neutron-dhcp-agent        |
| f098e91c-e805-44fc-8d9b-212446b90088 | DHCP agent           | controller1        | nova              | :-)   | True           | neutron-dhcp-agent        |

 

neutron dhcp-agent-list-hosting-net 9d2f02e1-5877-4f0a-a5b0-c18fc77018e4
+--------------------------------------+-------------+----------------+-------+
| id                                   | host        | admin_state_up | alive |
+--------------------------------------+-------------+----------------+-------+
| 60c33247-ffcd-4a0a-bb2f-75535ff56f75 | controller3 | True           | :-)   |
| 84a54e6d-f7d5-4f25-910b-5d025aa6aa3b | controller2 | True           | :-)   |
+--------------------------------------+-------------+----------------+-------+
neutron net-list-on-dhcp-agent 60c33247-ffcd-4a0a-bb2f-75535ff56f75 | grep 9d2f02e1-5877-4f0a-a5b0-c18fc77018e4
| 9d2f02e1-5877-4f0a-a5b0-c18fc77018e4 | nasi_weier                  | 29823194-5006-4005-8c98-9efd30742e1f 14.14.14.0/24 |
(neutron-dhcp-agent)[neutron@controller2 9d2f02e1-5877-4f0a-a5b0-c18fc77018e4]$ cat leases 1595494258 ac:8d:34:89:ae:ec 14.14.14.4 host-14-14-14-4 *
1595491072 fa:16:3e:14:b8:79 14.14.14.2 host-14-14-14-2 *
1595491072 fa:16:3e:03:42:55 14.14.14.3 host-14-14-14-3 *
1595498290 ac:8d:34:89:b1:5c 14.14.14.7 nasi *

 

root@controller2:/var/lib/docker/volumes/kolla_logs/_data/neutron# ip netns exec qdhcp-9d2f02e1-5877-4f0a-a5b0-c18fc77018e4 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ns-2aa0f452-75@if1088: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether fa:16:3e:03:42:55 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 14.14.14.3/24 brd 14.14.14.255 scope global ns-2aa0f452-75
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe03:4255/64 scope link 
       valid_lft forever preferred_lft forever
root@controller2:/var/lib/docker/volumes/kolla_logs/_data/neutron# ip netns exec qdhcp-9d2f02e1-5877-4f0a-a5b0-c18fc77018e4 ps -elf | grep dnsmasq | grep 9d2f02e1-5877-4f0a-a5b0-c18fc77018e4
5 S nobody    6419 36367  0  80   0 -  2305 poll_s 15:57 ?        00:00:00 dnsmasq --no-hosts --no-resolv --except-interface=lo --pid-file=/var/lib/neutron/dhcp/9d2f02e1-5877-4f0a-a5b0-c18fc77018e4/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/9d2f02e1-5877-4f0a-a5b0-c18fc77018e4/host --addn-hosts=/var/lib/neutron/dhcp/9d2f02e1-5877-4f0a-a5b0-c18fc77018e4/addn_hosts --dhcp-optsfile=/var/lib/neutron/dhcp/9d2f02e1-5877-4f0a-a5b0-c18fc77018e4/opts --dhcp-leasefile=/var/lib/neutron/dhcp/9d2f02e1-5877-4f0a-a5b0-c18fc77018e4/leases --dhcp-match=set:ipxe,175 --bind-interfaces --interface=ns-2aa0f452-75 --dhcp-range=set:tag0,14.14.14.0,static,86400s --dhcp-option-force=option:mtu,1500 --dhcp-lease-max=256 --conf-file=/etc/neutron/dnsmasq.conf --domain=openstacklocal
root@controller2:/var/lib/docker/volumes/kolla_logs/_data/neutron#

 

root@controller3:~# ip netns exec qdhcp-9d2f02e1-5877-4f0a-a5b0-c18fc77018e4 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ns-5e71e4c4-54@if737: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether fa:16:3e:14:b8:79 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 14.14.14.2/24 brd 14.14.14.255 scope global ns-5e71e4c4-54
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe14:b879/64 scope link 
       valid_lft forever preferred_lft forever
root@controller3:~# ip netns exec qdhcp-9d2f02e1-5877-4f0a-a5b0-c18fc77018e4 ps -elf | grep dnsmasq | grep 9d2f02e1-5877-4f0a-a5b0-c18fc77018e4
5 S nobody    3882  9633  0  80   0 -  2305 poll_s 15:57 ?        00:00:00 dnsmasq --no-hosts --no-resolv --except-interface=lo --pid-file=/var/lib/neutron/dhcp/9d2f02e1-5877-4f0a-a5b0-c18fc77018e4/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/9d2f02e1-5877-4f0a-a5b0-c18fc77018e4/host --addn-hosts=/var/lib/neutron/dhcp/9d2f02e1-5877-4f0a-a5b0-c18fc77018e4/addn_hosts --dhcp-optsfile=/var/lib/neutron/dhcp/9d2f02e1-5877-4f0a-a5b0-c18fc77018e4/opts --dhcp-leasefile=/var/lib/neutron/dhcp/9d2f02e1-5877-4f0a-a5b0-c18fc77018e4/leases --dhcp-match=set:ipxe,175 --bind-interfaces --interface=ns-5e71e4c4-54 --dhcp-range=set:tag0,14.14.14.0,static,86400s --dhcp-option-force=option:mtu,1500 --dhcp-lease-max=256 --conf-file=/etc/neutron/dnsmasq.conf --domain=openstacklocal
root@controller3:~#
(neutron-dhcp-agent)[neutron@controller3 9d2f02e1-5877-4f0a-a5b0-c18fc77018e4]$ ls
addn_hosts  host  interface  leases  opts  pid
(neutron-dhcp-agent)[neutron@controller3 9d2f02e1-5877-4f0a-a5b0-c18fc77018e4]$ cat leases 1595495232 1c:ae:cb:eb:92:b6 14.14.14.6 host-14-14-14-6 *
1595491073 fa:16:3e:14:b8:79 14.14.14.2 host-14-14-14-2 *
1595491073 fa:16:3e:03:42:55 14.14.14.3 host-14-14-14-3 *
1595498290 ac:8d:34:89:b1:5c 14.14.14.7 nasi 

 

 

本节介绍如何对DHCP代理可扩展性和HA使用代理管理(别名代理)和调度程序(别名代理调度程序)扩展。
使用neutron ext-list client命令检查是否启用了这些扩展。 检查代理和agent_scheduler包含在输出中。

$ neutron ext-list -c name -c alias
+-----------------+--------------------------+
| alias           | name                     |
+-----------------+--------------------------+
| agent_scheduler | Agent Schedulers         |
| binding         | Port Binding             |
| quotas          | Quota management support |
| agent           | agent                    |
| provider        | Provider Network         |
| router          | Neutron L3 Router        |
| lbaas           | Load Balancing service   |
| extraroute      | Neutron Extra Route      |
+-----------------+--------------------------+

设置演示
这里写图片描述

主机    描述
OpenStack controller host - controlnode    运行部署虚拟机所需的网络,身份和计算服务。 节点必须至少有一个网络接口连接到管理网络。 注意nova网络不应该运行,因为它被Neutron替代。
HostA    运行nova-compute, the Neutron L2 agent and DHCP agent
HostB    同HostA
配置
controlnode: neutron server
1.Neutron 配置文件 /etc/neutron/neutron.conf:

[DEFAULT]
core_plugin = linuxbridge
rabbit_host = controlnode
allow_overlapping_ips = True
host = controlnode
agent_down_time = 5
dhcp_agents_per_network = 1

在上面的配置中,我们使用dhcp_agents_per_network = 1进行演示。 在常规部署中,我们建议将dhcp_agents_per_network设置为多个以匹配部署中DHCP代理的数量。

2.更新 plug-in 配置文件 /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini:

[vlans]
tenant_network_type = vlan
network_vlan_ranges = physnet1:1000:2999
[database]
connection = mysql://root:root@127.0.0.1:3306/neutron_linux_bridge
retry_interval = 2
[linux_bridge]
physical_interface_mappings = physnet1:eth0

HostA and HostB: L2 agent
1.Neutron 配置文件 /etc/neutron/neutron.conf:

[DEFAULT]
rabbit_host = controlnode
rabbit_password = openstack
# host = HostB on hostb
host = HostA

5
2.更新 plug-in 配置文件 /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini:

[vlans]
tenant_network_type = vlan
network_vlan_ranges = physnet1:1000:2999
[database]
connection = mysql://root:root@127.0.0.1:3306/neutron_linux_bridge
retry_interval = 2
[linux_bridge]
physical_interface_mappings = physnet1:eth0

3.更新 the nova 配置文件 /etc/nova/nova.conf:

[DEFAULT]
use_neutron=True
firewall_driver=nova.virt.firewall.NoopFirewallDriver

[neutron]
admin_username=neutron
admin_password=servicepassword
admin_auth_url=http://controlnode:35357/v2.0/
auth_strategy=keystone
admin_tenant_name=servicetenant
url=http://100.1.1.10:9696/

11
HostA and HostB: DHCP agent
更新 DHCP 配置文件 /etc/neutron/dhcp_agent.ini:

[DEFAULT]
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver
1
2
演示的先决条件
需要管理员角色才能使用代理管理和调度程序扩展。 确保在具有管理员角色的项目下运行以下命令。
要进行实验,您需要VM和Neutron网络:

$ nova list
+--------------------------------------+-----------+--------+---------------+
| ID                                   | Name      | Status | Networks      |
+--------------------------------------+-----------+--------+---------------+
| c394fcd0-0baa-43ae-a793-201815c3e8ce | myserver1 | ACTIVE | net1=10.0.1.3 |
| 2d604e05-9a6c-4ddb-9082-8a1fbdcc797d | myserver2 | ACTIVE | net1=10.0.1.4 |
| c7c0481c-3db8-4d7a-a948-60ce8211d585 | myserver3 | ACTIVE | net1=10.0.1.5 |
+--------------------------------------+-----------+--------+---------------+

$ neutron net-list
+--------------------------------------+------+--------------------------------------+
| id                                   | name | subnets                              |
+--------------------------------------+------+--------------------------------------+
| 89dca1c6-c7d4-4f7a-b730-549af0fb6e34 | net1 | f6c832e3-9968-46fd-8e45-d5cf646db9d1 |
+--------------------------------------+------+--------------------------------------+

管理Neutron部署中的代理
1.列出所有代理

$ neutron agent-list
+--------------------------------------+--------------------+-------+-------+----------------+
| id                                   | agent_type         | host  | alive | admin_state_up |
+--------------------------------------+--------------------+-------+-------+----------------+
| 1b69828d-6a9b-4826-87cd-1757f0e27f31 | Linux bridge agent | HostA | :-)   | True           |
| a0c1c21c-d4f4-4577-9ec7-908f2d48622d | DHCP agent         | HostA | :-)   | True           |
| ed96b856-ae0f-4d75-bb28-40a47ffd7695 | Linux bridge agent | HostB | :-)   | True           |
| f28aa126-6edb-4ea5-a81e-8850876bc0a8 | DHCP agent         | HostB | :-)   | True           |
+--------------------------------------+--------------------+-------+-------+----------------+

每个支持这些扩展的代理在启动时都会向neutron服务器注册。
输出显示四个代理的信息。 活动字段显示:-)如果代理在neutron.conf文件中的agent_down_time选项定义的时间段内报告其状态。 否则活着是xxx。

2.列出托管指定网络的DHCP代理:

$ neutron dhcp-agent-list-hosting-net net1
+--------------------------------------+-------+----------------+-------+
| id                                   | host  | admin_state_up | alive |
+--------------------------------------+-------+----------------+-------+
| a0c1c21c-d4f4-4577-9ec7-908f2d48622d | HostA | True           | :-)   |
+--------------------------------------+-------+----------------+-------+

3.列出由给定DHCP代理托管的网络:
此命令将显示给定dhcp代理正在管理的网络。

$ neutron net-list-on-dhcp-agent a0c1c21c-d4f4-4577-9ec7-908f2d48622d
+--------------------------------------+------+--------------------------------------------------+
| id                                   | name | subnets                                          |
+--------------------------------------+------+--------------------------------------------------+
| 89dca1c6-c7d4-4f7a-b730-549af0fb6e34 | net1 | f6c832e3-9968-46fd-8e45-d5cf646db9d1 10.0.1.0/24 |
+--------------------------------------+------+--------------------------------------------------+

4.显示代理详细信息。
neutron agent-show命令显示指定代理程序的详细信息:

$ neutron agent-show a0c1c21c-d4f4-4577-9ec7-908f2d48622d
+---------------------+--------------------------------------------------------+
| Field               | Value                                                  |
+---------------------+--------------------------------------------------------+
| admin_state_up      | True                                                   |
| agent_type          | DHCP agent                                             |
| alive               | False                                                  |
| binary              | neutron-dhcp-agent                                     |
| configurations      | {                                                      |
|                     |     "subnets": 1,                                      |
|                     |     "dhcp_driver": "neutron.agent.linux.dhcp.Dnsmasq", |
|                     |     "networks": 1,                                     |
|                     |     "dhcp_lease_time": 120,                            |
|                     |     "ports": 3                                         |
|                     | }                                                      |
| created_at          | 2013-03-16T01:16:18.000000                             |
| description         |                                                        |
| heartbeat_timestamp | 2013-03-17T01:37:22.000000                             |
| host                | HostA                                                  |
| id                  | 58f4ce07-6789-4bb3-aa42-ed3779db2b03                   |
| started_at          | 2013-03-16T06:48:39.000000                             |
| topic               | dhcp_agent                                             |
+---------------------+--------------------------------------------------------+

23
在此输出中,heartbeat_timestamp是neutron服务器上的时间。 此扩展程序无需同步所有代理程序,即可正常运行。 配置描述了代理程序或运行时数据的静态配置。 此代理是DHCP代理,它承载一个网络,一个子网和三个端口。
不同类型的代理显示不同的细节。 以下输出显示了Linux网桥代理的信息:

$ neutron agent-show ed96b856-ae0f-4d75-bb28-40a47ffd7695
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| admin_state_up      | True                                 |
| binary              | neutron-linuxbridge-agent            |
| configurations      | {                                    |
|                     |      "physnet1": "eth0",             |
|                     |      "devices": "4"                  |
|                     | }                                    |
| created_at          | 2013-03-16T01:49:52.000000           |
| description         |                                      |
| disabled            | False                                |
| group               | agent                                |
| heartbeat_timestamp | 2013-03-16T01:59:45.000000           |
| host                | HostB                                |
| id                  | ed96b856-ae0f-4d75-bb28-40a47ffd7695 |
| topic               | N/A                                  |
| started_at          | 2013-03-16T06:48:39.000000           |
| type                | Linux bridge agent                   |
+---------------------+--------------------------------------+

21
管理网络到DHCP代理的分配
单个网络可以分配给多个DHCP代理,一个DHCP代理可以承载多个网络。 您可以向DHCP代理添加网络,并从中删除一个。
1.默认计划。
当您使用一个端口创建网络时,网络将被调度为活动的DHCP代理。 如果许多活动DHCP代理正在运行,请随机选择一个。 您可以设计更复杂的调度算法与nova-schedule相同的方式。

$ neutron net-create net2
$ neutron subnet-create net2 9.0.1.0/24 --name subnet2
$ neutron port-create net2
$ neutron dhcp-agent-list-hosting-net net2
+--------------------------------------+-------+----------------+-------+
| id                                   | host  | admin_state_up | alive |
+--------------------------------------+-------+----------------+-------+
| a0c1c21c-d4f4-4577-9ec7-908f2d48622d | HostA | True           | :-)   |
+--------------------------------------+-------+----------------+-------+

它分配给HostA上的DHCP代理。 如果要通过dnsmasq命令验证行为,则必须为网络创建子网,因为DHCP代理仅在存在DHCP时启动dnsmasq服务。

2.为DHCP代理分配网络
要添加另一个DHCP代理来托管网络,请运行以下命令:

$ neutron dhcp-agent-network-add f28aa126-6edb-4ea5-a81e-8850876bc0a8 net2
Added network net2 to dhcp agent
$ neutron dhcp-agent-list-hosting-net net2
+--------------------------------------+-------+----------------+-------+
| id                                   | host  | admin_state_up | alive |
+--------------------------------------+-------+----------------+-------+
| a0c1c21c-d4f4-4577-9ec7-908f2d48622d | HostA | True           | :-)   |
| f28aa126-6edb-4ea5-a81e-8850876bc0a8 | HostB | True           | :-)   |
+--------------------------------------+-------+----------------+-------+

9
两个DHCP代理都托管net2网络。

3.从指定的DHCP代理中删除网络。
此命令是上一个命令的同级命令。 从HostA的DHCP代理中删除net2:

$ neutron dhcp-agent-network-remove a0c1c21c-d4f4-4577-9ec7-908f2d48622d   net2
Removed network net2 to dhcp agent
$ neutron dhcp-agent-list-hosting-net net2
+--------------------------------------+-------+----------------+-------+
| id                                   | host  | admin_state_up | alive |
+--------------------------------------+-------+----------------+-------+
| f28aa126-6edb-4ea5-a81e-8850876bc0a8 | HostB | True           | :-)   |
+--------------------------------------+-------+----------------+-------+

可以看到只有HostB的DHCP代理托管net2网络。

DHCP代理的HA
在net2上引导虚拟机。 让两个DHCP代理主机net2。 依次失败代理查看VM是否仍然可以获得所需的IP。
1.在net2上启动虚拟机

$ neutron net-list
+--------------------------------------+------+--------------------------------------------------+
| id                                   | name | subnets                                          |
+--------------------------------------+------+--------------------------------------------------+
| 89dca1c6-c7d4-4f7a-b730-549af0fb6e34 | net1 | f6c832e3-9968-46fd-8e45-d5cf646db9d1 10.0.1.0/24 |
| 9b96b14f-71b8-4918-90aa-c5d705606b1a | net2 | 6979b71a-0ae8-448c-aa87-65f68eedcaaa 9.0.1.0/24  |
+--------------------------------------+------+--------------------------------------------------+
$ nova boot --image tty --flavor 1 myserver4   --nic net-id=9b96b14f-71b8-4918-90aa-c5d705606b1a
...
$ nova list
+--------------------------------------+-----------+--------+---------------+
| ID                                   | Name      | Status | Networks      |
+--------------------------------------+-----------+--------+---------------+
| c394fcd0-0baa-43ae-a793-201815c3e8ce | myserver1 | ACTIVE | net1=10.0.1.3 |
| 2d604e05-9a6c-4ddb-9082-8a1fbdcc797d | myserver2 | ACTIVE | net1=10.0.1.4 |
| c7c0481c-3db8-4d7a-a948-60ce8211d585 | myserver3 | ACTIVE | net1=10.0.1.5 |
| f62f4731-5591-46b1-9d74-f0c901de567f | myserver4 | ACTIVE | net2=9.0.1.2  |
+--------------------------------------+-----------+--------+---------------+

19
确保两个托管net2的DHCP代理:
使用先前的命令将网络分配给代理。

$ neutron dhcp-agent-list-hosting-net net2
+--------------------------------------+-------+----------------+-------+
| id                                   | host  | admin_state_up | alive |
+--------------------------------------+-------+----------------+-------+
| a0c1c21c-d4f4-4577-9ec7-908f2d48622d | HostA | True           | :-)   |
| f28aa126-6edb-4ea5-a81e-8850876bc0a8 | HostB | True           | :-)   |
+--------------------------------------+-------+----------------+-------+


要测试DHCP代理的HA:
1.登录myserver4虚拟机,执行udhcpc,dhclient或其他DHCP客户端。
2.停止HostA上的DHCP代理。 除了停止neutron-dhcp-agent二进制文件,必须停止dnsmasq进程。
3.在虚拟机中运行DHCP客户端,以查看它是否可以获得所需的IP。
4.停止HostB上的DHCP代理。
5.在VM中运行udhcpc; 它不能得到想要的IP。
6.在HostB上启动DHCP代理。 VM再次获取所需的IP。

禁用删除一个代理
如果计划进行系统硬件或软件升级,管理员可能需要禁用代理。 一些支持调度的代理还支持禁用和启用代理,例如L3和DHCP代理。 禁用代理后,调度程序不会向代理调度新资源。
禁用代理后,您可以安全地删除代理。 即使在禁用代理后,也会继续分配代理上的资源。 在删除代理之前,请确保删除代理上的资源。
在停止HostA之前禁用HostA上的DHCP代理:

$ neutron agent-update a0c1c21c-d4f4-4577-9ec7-908f2d48622d --admin-state-up False
$ neutron agent-list
+--------------------------------------+--------------------+-------+-------+----------------+
| id                                   | agent_type         | host  | alive | admin_state_up |
+--------------------------------------+--------------------+-------+-------+----------------+
| 1b69828d-6a9b-4826-87cd-1757f0e27f31 | Linux bridge agent | HostA | :-)   | True           |
| a0c1c21c-d4f4-4577-9ec7-908f2d48622d | DHCP agent         | HostA | :-)   | False          |
| ed96b856-ae0f-4d75-bb28-40a47ffd7695 | Linux bridge agent | HostB | :-)   | True           |
| f28aa126-6edb-4ea5-a81e-8850876bc0a8 | DHCP agent         | HostB | :-)   | True           |
+--------------------------------------+--------------------+-------+-------+----------------+

在HostA上停止DHCP代理后,可以通过以下命令删除它:

$ neutron agent-delete a0c1c21c-d4f4-4577-9ec7-908f2d48622d
Deleted agent: a0c1c21c-d4f4-4577-9ec7-908f2d48622d
$ neutron agent-list
+--------------------------------------+--------------------+-------+-------+----------------+
| id                                   | agent_type         | host  | alive | admin_state_up |
+--------------------------------------+--------------------+-------+-------+----------------+
| 1b69828d-6a9b-4826-87cd-1757f0e27f31 | Linux bridge agent | HostA | :-)   | True           |
| ed96b856-ae0f-4d75-bb28-40a47ffd7695 | Linux bridge agent | HostB | :-)   | True           |
| f28aa126-6edb-4ea5-a81e-8850876bc0a8 | DHCP agent         | HostB | :-)   | True           |
+--------------------------------------+--------------------+-------+-------+----------------+

默认开启DHCP高可用性
您可以通过在文件/etc/neutron/neutron.conf中设置以下配置选项来控制分配给网络的DHCP代理的默认数量。

dhcp_agents_per_network = 3

 

neutron dhcp 高可用

标签:word   root   compute   default   mes   out   停止   执行   节点   

原文地址:https://www.cnblogs.com/dream397/p/13362477.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!